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FOREWORD 


This  report  documents  the  methodology  and  lessons  learned  in  developing  the  Innovative 
Tools  and  Techniques  for  Brigade  and  Below  Staff  Training  Program’s  Staff  Group  Trainer 
(SGT)  Project.  Increasing  turbulence  and  inexperience  in  brigade  staffs,  coupled  with  the  high 
demands  on  training  resources,  create  a  significant  training  challenge  for  commanders.  This 
training  project  provides  the  commander  a  tool  to  meet  this  challenge.  The  project  is  part  of  the 
continuing  research  to  establish  innovative  methods  for  training  combined  arms  forces  by  the 
U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences.  This  SGT  project  was 
designed  to  meet  the  specific  training  needs  for  brigade  staff  functions  that  support  the 
commander.  This  effort  refines  past  SGT  work,  while  streamlining  and  reducing  the 
requirements  for  overhead  training  assistance. 

The  outcome  of  this  SGT  project  provides  training  for  armored  force  brigade  staffs  to 
support  their  commander’s  decision  making  process  in  the  command  and  control  cycle.  The 
structured  exercises  challenge  the  staff  to  perform  its  functions  in  the  execution  phase  of  a  unit 
mission.  The  training  project  is  designed  for  a  newly  formed,  inexperienced  staff.  The  project 
bridges  the  training  gap  between  individual  staff  member  skills  and  complex  collective  staff 
skills,  and  prepares  a  staff  for  more  complex  simulation-based  or  field  training  exercises. 

The  newly  developed  features  and  automated  training  support  packages  of  this  project  are 
directed  at  providing  an  easy-to-use  system  that  minimizes  the  requirements  for  preparation  and 
maximizes  the  training  value  for  the  commander  and  his  staff. 


ZITAM.  SIMUTIS 
Technical  Director 


V 


ACKNOWLEDGMENTS 


This  report  reflects  the  contributions  of  a  dedicated  team  of  support  personnel  providing 
expertise  in  military  content,  instructional  system  design,  formative  evaluation,  technical 
support,  and  administration.  The  authors  would  like  to  acknowledge  the  following  BDM  team 
members:  Daniel  B.  Britt,  Kenneth  W.  Fergus,  David.  O.  Johnson,  Barry  M.  Quensel,  Kathy  C. 
Smallwood,  Victoria  L.  Winemiller,  Linda  Clark,  and  Lori  L.  Bailey.  Thanks  also  go  to  Joseph 
Cassidy  (PRC)  for  his  contributions. 

Overall  supervision  of  the  project  was  provided  initially  by  Robert  S.  Sever,  who  was 
succeeded  by  Dallas  L.  Long  (both  of  BDM).  The  Senior  Scientist,  Dr.  Bruce  C.  Leibrecht 
(BDM),  contributed  many  hours  in  reviewing  the  project’s  design  and  report  documentation;  his 
efforts  significantly  contributed  to  keeping  this  effort  on  track.  Mitchell  S.  Greess  and  Robert 
Bogardus,  software  development  sub-contractors,  and  Ronald  L.  Mast  (BDM)  were  responsible 
for  a  significant  code  writing  effort  that  made  the  training  system  a  reality.  Dr.  Kathleen  A, 
Quinkert  (Army  Research  Institute)  served  as  the  Contracting  Officer’s  Representative  and 
expertly  guided  the  Team’s  efforts. 

Finally,  special  thanks  go  to  the  Mounted  Warfare  Test  Bed  for  technical  support;  to 
members  of  the  User’s  Panel  for  invaluable  input;  and  to  the  Kentucky  Army  National  Guard 
(ARNG)  and  Texas  ARNG,  whose  participation  in  the  pilot  and  implementation  trial  helped 
demonstrate  the  training  program’s  capabilities. 


VI 


DEVELOPMENT  OF  A  REFINED  STAFF  GROUP  TRAINER 


EXECUTIVE  SUMMARY 


Research  Requirement: 

This  is  a  crucial  time  in  the  U.S.  Army  where  decreasing  defense  budgets,  turbulence,  and 
the  increasing  complexity  of  advanced  battlefield  technology  impact  training  adequacy  across 
most  staff  areas.  Units  need  training  to  be  less  resource  intensive  and  more  focused  on  the  skills 
they  most  critically  need  to  achieve  true  fighting  proficiency.  This  is  especially  true  for  staffs, 
where  day  to  day  operations  leave  little  room  for  battle  staff  training. 

The  objective  of  this  Staff  Group  Trainer  (SGT)  research  and  development  project  was  to 
refine  a  previously  developed,  computer-driven,  structured  military  staff  training  simulation 
system  that  concentrated  on  training  at  the  brigade  staff  level.  The  project  was  based  on  a  staff 
process  model  and  designed  to  develop  exercises  focused  on  specific  staff  learning  objectives. 
These  exercises  were  based  on  critical,  doctrinally  important  activities  within  and  between  staff 
sections  in  the  brigade  main  command  post. 

Procedure: 

The  Team'  conducted  a  needs  assessment  with  selected  brigade  level  commanders  to 
determine  the  precise  nature  of  the  training  gap  between  individual  staff  member  training  and 
complex  collective  staff  training  for  brigade  staffs.  Using  the  analysis  of  this  information  as  a 
starting  point  and  guide,  the  Team  used  the  previously  developed  staff  training  project  concept 
from  past  SGT  work  as  a  strawman  to  refine  and  develop  a  new  exercise  library  for  a  set  of 
brigade  staff  exercises. 

Next,  the  Team  reexamined  the  process  by  which  exercises  were  developed.  Standardized 
worksheets  were  created  to  guide  the  exercise  development  process.  Using  the  Area  Defense 
mission,  the  Team  reanalyzed  the  tactical  scenario  and  refined  the  library  for  that  mission  using  a 
three-level  (crawl-walk-run)  approach,  resulting  in  tables  increasing  in  complexity. 

The  project  followed  an  iterative  research  and  development  process.  Internal  testing  and 
refinement  of  the  exercises  were  followed  by  a  pilot,  then  a  trial.  At  each  step,  refinements  and 
modifications  were  made  to  the  materials  and  the  system  based  on  the  participants’  comments. 
The  participants  were  actual  intact  staffs,  drawn  from  the  National  Guard. 

Findings: 

The  needs  assessment  identified  the  appropriate  training  focus  for  the  exercises,  and  the 
multimedia  previews  and  train-the-trainer  packages  demonstrated  the  ability  to  operate  and 


'  The  word  “Team”  will  be  used  throughout  the  document  to  refer  to  the  SGT  Research  Group 
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evaluate  the  training  without  external  training  support  personnel. 

The  formative  evaluation  showed  that  most  participants  found  the  exercises  appropriately 
challenging,  focused  on  the  correct  tasks,  and  achieving  their  training  objectives.  Training 
feedback,  in  the  form  of  after  action  reviews  (AARs),  at  both  section  and  command  post  levels, 
was  rated  as  effective  by  most  haining  participants. 

Utilization  of  Findings: 

The  refined  SGT  system  represents  a  necessary  link  in  an  integrated  staff  training  cycle  that 
moves  all  staff  members  from  initial  individual  proficiency  to  a  more  proficient  state  of  combat 
readiness.  New  research  should  continue  to  demonstrate  the  contribution  of  the  SGT  to  staff 
command  and  control  with  particular  attention  paid  to  the  development  of  shared  mental  models. 
Additional  research  should  also  address  multimedia  and  technical  innovations  specifically 
focused  on  improving  the  value  of  the  refined  SGT  system  to  the  U.S.  Army.  The  SGT  has 
potential  application  to  training  for  any  problem  solving  situation  which  is  time-critical  and 
involves  interdependent  relationships  between  dissimilar  staff  agencies,  as  can  be  found  in 
disaster  or  crisis  management  situations  involving  natural  (hurricanes,  floods,  fires)  or  man-made 
(war,  terrorism,  nuclear  or  chemical  accidents)  circumstances. 

This  training  system  concept  needs  further  refinement  and  development  prior  to  full 
implementation.  Additional  work  is  needed  to  further  define  and  develop  this  project  into  a  more 
“user  fiiendly”  and  technically-compatible  (e.g.  common  hardware)  system  for  use  by 
commanders  in  the  field.  Additional  exercises  that  address  typical  missions,  functional  areas, 
and  digital  applications  should  be  created  to  meet  the  diverse  needs  of  field  commanders.  Train- 
up  time  should  be  reduced  through  the  refinement  of  the  system’s  current  computer-based 
instruction  technology. 
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DEVELOPMENT  OF  A  REFINED  STAFF  GROUP  TRAINER 


INTRODUCTION 

This  report  describes  the  Staff  Group  Trainer  (SGT)  portion  of  a  program  entitled  Innovative 
Tools  and  Techniques  for  Brigade  and  Below  Staff  Training  II  (ITTBBST II).  The  SGT  project 
was  a  research  and  development  effort  to  expand  and  refine  the  existing  structured  computer- 
aided  SGT.  Specific  program  objectives  included  the  following: 

1 .  Design  (refine)  the  existing  SGT  brigade  level  training  project. 

2.  Develop  a  sound,  small  group  (brigade  staff)  training  methodology. 

3.  Produce  sample  training  support  package  (TSP)  products. 

4.  Create  a  training  system  on  the  existing  training  platform. 

5.  Document  the  project. 

Earlier  SGT  work  demonstrated  the  structured  staff  training  methodology  and  its  support  of 
the  commander’s  decision  making  process.  Building  on  the  prior  SGT  efforts,  the  exercises  for 
the  current  effort  were  designed  to  be  progressively  more  difficult  (crawl,  walk,  run)  as  specified 
in  Army  training  policy  (Department  of  the  Army,  1988a)  and  contained  three  distinct  training 
levels  :  staff  transition  exercises  (training  within  a  staff  section),  staff  integration  exercises 
(training  involving  selected  sections  working  together  on  specific  situations),  and  command  post 
(CP)  exercises  (training  within  the  entire  CP,  [i.e.,  integrated  sections]).  This  report  documents 
the  methodology,  results,  and  lessons  learned  from  the  conduct  of  the  current  SGT  project. 

Background 

The  development  of  innovative  techniques  represents  a  major  focus  of  the  Force  XXI 
Training  Program,  which  is  designed  to  transform  the  Army’s  training  capabilities  in  the  21st 
century  (U.S.  Army  Training  and  Doctrine  Command  [TRADOC],  1994).  The  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  conducted  extensive  research  on 
military  performance  as  it  relates  to  training  (e.g.,  Holz,  Hiller,  &  McFann,  1994).  They  have 
collected  original  data  as  well  as  condensing  and  analyzing  volumes  of  combat  training  center 
(CTC)  data  in  an  effort  to  identify  those  aspects  of  training  which  impact  unit  performance.  One 
body  of  research,  begun  in  1989  under  the  auspices  of  the  Determinants  of  Effective  Unit 
Performance  Research  Program  (Holz  et  al.,  1994),  identified  deficiencies  in  training  staff 
synchronization  (Thompson,  Pleban,  &  Valentine,  1994)  and  determined  that  battle  staff 
operations  correlated  positively  with  unit  combat  success  (Keesling,  Ford,  &  Harrison,  1994). 
The  latter  group  identified  three  staff  constructs  which  correlated  significantly  with  force-on- 
force  success:  (a)  good  staff  standing  operating  procedures  (SOP)  and  training  on  the  SOP, 

(b)  staffs  which  could  issue  accurate  and  timely  orders,  and  (c)  good  staff  integration-i.e.,  staff 
obtaining,  sharing,  and  interpreting  information,  and  cooperating  with  one  another  (Keesling  et 
al.,  1994,  p.  145). 
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Most  units  at  the  National  Training  Center  (NTC)  did  not  demonstrate  proficiency  in  these 
staff  areas.  In  fact,  commanders  and  staff  officers  were  interviewed  and  “a  disproportionate 
number . . .  felt  they  were  less  than  fully  prepared  to  fill  staff  positions”  (Thompson  et  al.,  1994, 
p.  185).  This  feeling  of  lack  of  preparedness  was  not  without  foundation — ^there  was  little 
evidence  of  systematic  staff  functional  area  training  during  the  branch  oriented  phases  of  their 
professional  development.  Nor  did  key  staff  members  remain  together  for  long  enough  periods 
of  time  so  that  they  could  develop  into  a  highly  functional  staff  (Thompson  et  al.,  1994).  To 
overcome  this.  Brown  (1994)  recommended  a  new  training  paradigm  which  would  be  built 
around  repetitive  training  situations  designed  to  cue  specific  individual,  staff,  or  imit  behavior. 
According  to  Brown,  structured  modules  would  train  staffs  in  the  basics.  These  exercise 
modules  would  be  structured  to  provide  a  ‘seamless’  crawl,  walk,  run  progression  and  would  be 
followed  by  situational  training  exercises,  but  only  after  the  ‘basics’  had  been  mastered. 

With  the  downsizing  of  the  Army  over  the  last  several  years,  high  turnover  of  staffs  may  be 
assumed  to  be  worsening.  Given  the  cycle  of  unit  rotations  through  the  CTC,  it  is  realistic  to 
assume  that  a  given  staff  will  have  only  one  CTC  rotation  together.  These  training  audience 
characteristics  were  established  during  earlier  SGT  efforts,  based  on  a  projected  need  for  this 
level  of  training. 


Review  of  Preceding  Projects 

Of  particular  importance  to  the  current  SGT  design  was  the  work  which  addressed  structured 
staff  training  in  the  simulation-based  training  programs  established  for  the  Reserve  and  Active 
Components  and  individual  computer-based  training  (e.g.,  C.  Campbell,  R.  Campbell,  Sanders, 
Flynn,  &  Myers,  1995;  Graves,  Campbell,  Deter,  &  Quinkert,  1997;  Andre,  Wampler,  &  Olney, 
1997).  Also  of  interest  was  the  body  of  work  conducted  by  other  ARI  elements,  the  Naval  Air 
Warfare  Center  Training  Systems  Division  in  Orlando,  Florida  and  throughout  academia. 

This  project  reflects  the  progressive  development  of  an  innovative  training  approach 
designed  to  support  simulation-based  staff  training.  The  paragraphs  below  discuss  those 
individual  and  collective  training  programs  from  which  this  SGT  effort  evolved. 

Individual  Training  Program 

The  current  Battle  Staff  Training  System  (BSTS)  training  program  was  developed  as  an 
ITTBBST  project.  The  BSTS  training  program  provides  computer-assisted  instruction  on 
individual  staff  skills  for  conunanders  and  staff  officers  in  armored  and  mechanized  infantry 
battalions  and  brigades  (Andre  et  al.,  1997)  and  contains  training  packages  for  principal  and 
support  staff  officers  at  both  echelons.  It  provides  Army  officers  with  multimedia  courseware 
that  enables  them  to  enhance  their  individual  staff  skills.  This  project  also  served  as  a  source  for 
individual  remediation  training  within  the  current  effort. 
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Collective  Training  Programs 


General 


The  Simulation-Based  Multiechelon  Training  Program  for  Armor  Units  (SIMUTA), 
Simulation-Based  Mounted  Brigade  Training  Program  (SIMBART),  and  Combined-arms 
Operations  at  Brigade  Level,  Realistically  Achieved  through  Simulation  (COBRAS)  programs 
represent  the  collective  training  evolution  of  the  SGT  design.  These  programs  are  discussed  in 
chronological  order. 

SIMUTA 

The  SIMUTA  program  was  developed  by  leveraging  existing  and  emerging  technologies  for 
training  (Hoffinan,  Graves,  Roger,  Flynn,  &  Sever,  1995).  It  represented  an  early  effort  to 
develop  structured  training  and  included:  (a)  standardized  exercises,  (b)  missions  divided  into 
structured  tables  which  took  approximately  one  hour  to  execute,  (c)  clearly  defined  unit 
performance  expectations  for  each  table,  and  (d)  after  action  reviews  (AARs)  designed  to  follow 
each  table.  Key  concepts  of  structured  training  include  identifying  tasks  or  processes  to  be 
trained,  using  a  scenario  containing  events  which  will  cue  the  tasks  or  processes  to  be  trained, 
and  providing  feedback  on  task  or  process  performance. 

The  SIMUTA  program  focused  on  ARNG  battalion  and  below  training  with  the  focus  on 
training  the  execution  phase  of  combat.  It  was  specifically  designed  to  use  the  Simulation 
Networking  (SIMNET)  virtual  environment  for  platoon  through  battalion  level  training  and  the 
Janus  constructive  environment  and  workstations  from  the  earlier  ARI  Combat  Vehicle 
Command  and  Control  (CVCC)  project  for  battalion  staff  training  (Hofi&nan,  1993).  Three 
different  types  of  exercises  were  developed — ^Janus,  SIMNET,  and  Commander/Staff  Trainer 
(C/ST)  exercises.  The  SIMNET  and  Janus  exercises  in  SIMUTA  were  built  around  two  NTC 
missions  (movement  to  contact  [MTC]  and  defense)  while  the  C/ST  exercises  used  only  the 
MTC  mission  (Hofi&nan,  1993).  The  C/ST’s  role  in  SIMUTA  was  to  provide  staff  officers  with 
training  in  processing  tactically  relevant  information  using  replicable,  computer-driven  exercises. 

SIMBART  and  COBRAS  I 


The  next  two  programs  were  developmentally  intertwined.  Both  SIMBART  and  COBRAS  I 
used  the  same  operations  orders  (OPORDs)  and  missions;  however,  they  had  different  training 
foci.  The  SIMBART  focused  on  training  a  heavy  brigade  staff  in  the  exeeution  phase  of  the 
NTC  defensive  and  offensive  (MTC  and  attack)  missions  (Koger  et  al.,  1996).  The  COBRAS  I 
contained  training  on  all  three  phases  (plan,  prepare,  execute)  although  roughly  80%  of  the 
training  focused  on  the  plan  and  prepare  phases,  so  the  execution  phase  was  not  as  heavily 
weighted  in  this  program  (Graves,  Campbell,  Deter,  &  Quinkert,  1997).  The  SIMBART 


*  COBRAS  I  was  the  first  in  a  series  of  projects  bearing  the  COBRAS  name.  COBRAS  I  was  conducted  from 
January  1995  through  May  1996.  COBRAS  II  was  conducted  from  October  1995  -  March  1997.  COBRAS  III  was 
conducted  from  April  1997  through  May  1998. 
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program  design  applied  the  structured  training  methodology  to  the  brigade  staff.  Just  as  the 
SIMUTA  lessons  learned  (Hoffman  et  al.,  1995)  influenced  the  SIMBART  development,  the 
SIMBART  lessons  learned  (Koger  et  al.,  1996)  indicated  potential  design  changes  for 
consideration  in  the  SGT  project. 

The  SIMBART  program  provided  significant  insight  into  brigade  level  staff  training  (Koger 
et  al.,  1996).  The  most  relevant  issues  to  the  SGT  projects  were: 

1 .  Staff  actions  (especially  intra-staff  activity)  are  not  well  codified  in  military  doctrine. 

2.  AARs  need  to  focus  on  staff  support  for  the  commander  rather  than  on  tactical 
outcomes. 

3.  There  is  a  need  for  standardized  coaching  questions  (keyed  to  exercise  learning 
objectives)  for  observers  to  guide  staff  members  during  the  exercise. 

Staff  Group  Trainer 

The  most  recent  SGT  effort  hosted  structured  training  on  a  computer  network  which  could 
automatically  collect  information  on  certain  trainee  behaviors.  Drawing  from  the  foundation 
established  during  the  SIMUTA  C/ST  effort  (Koger  et  al.,  1998),  the  Team  for  the  prior  effort 
developed  a  limited  set  of  brigade  and  battalion-level  structured  staff  training  exercises  on  the 
execution  phase  of  three  NTC  missions  (MTC,  defense,  and  attack).  The  exercises  in  the  stages 
were  sequenced  to  increase  in  complexity  (i.e.,  crawl,  walk,  run  difficulty  levels)  as  trainees 
completed  the  tables,  working  towards  the  automation  of  staff  skills.  Changes  in  the  prior  SGT 
TSPs  were  based  on  SIMUTA/SIMBART  lessons  learned.  Lessons  learned  during  the  prior 
SGT  effort  also  shaped  development  of  the  current  effort.  Key  lessons  learned  from  each  of 
these  efforts  are  described  below. 

One  lesson  learned  relates  to  Olmstead’s  modification  of  Schein’s  (1965)  adaptive  coping 
cycle  for  organizational  processes.  Though  Olmstead  (1992)  adapted  the  cycle  specifically  to 
identify,  isolate,  and  evaluate  military  staff  processes;  the  SIMBART  team  found  it  necessary  to 
extend  the  terminology  from  Olmstead’s  academic  terms  to  doctrinally-related  Army  terms 
which  became  the  basis  of  the  structured  AARs  (Koger  et.  al,  1998).  These  SIMBART  terms 
(i.e.,  monitor,  process,  analyze/evaluate,  communicate,  coordinate,  integrate,  recommend, 
disseminate,  and  synchronize)  formed  the  basis  from  which  the  prior  SGT  program’s  learning 
objectives  were  developed.  A  second  lesson  learned  demonstrated  the  importance  of 
incorporating  coaching  question  guidelines  into  observer  checklists  (Koger  et  al.,  1998).  This 
approach  was  adopted  for  the  current  effort  with  questions  designed  to  keep  the  unit  staff 
members  focused  on  critical  elements  of  the  exercise.  Lessons  learned  from  SIMBART  also 
highlighted  the  importance  of  structuring  AARs  around  staff  performance.  Thus,  the  AAR 
design  for  the  SGT  efforts  made  a  more  focused  attempt  than  SIMBART  to  structure  AARs 
around  staff  performance.  The  observer/controllers  (0/Cs)  were  provided  prepared  slides  and  a 
prototype  multimedia  presentation  to  facilitate  concentration  on  exercise  learning  objectives 
(Koger  et  al.,  1998).  A  final  key  lesson  learned  from  the  prior  SGT  effort  concerned  the  focus  of 
training.  Comments  received  during  pilot  testing  and  unit  trials  suggested  that  the  training 
audience  design  characteristics  were  incorrect  (Koger  et  al.,  1998).  Pilot  participants  expressed 
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concerns  that  the  training  focus  for  the  earlier  SGT  effort  was  inappropriate  for  the  target 
audience.  The  training  developers  also  gained  insights  during  the  development  cycle,  some  of 
which  were  incorporated  into  the  present  program. 

Problem  Statement 

A  variety  of  Army  programs  support  training  of  brigade  and  battalion  staffs.  However,  for 
the  successful  training  of  unit  staffs,  training  should  proceed  in  a  structured  maimer  from 
individual  to  small  staff  groups,  then  to  the  integrated  staff,  and  finally  to  the  full  unit  level  (Hall, 
Dwyer,  Cannon-Bowers,  Salas,  &  Volpe,  1993).  At  this  point  there  is  a  serious  shortage  of 
training  tools  for  units  to  use  in  training  small  staff  groups.  Without  tools  to  bridge  the  gap  from 
individual  competence  to  proficiency  of  full  staffs,  brigades  and  battalions  are  hampered  in 
meeting  their  training  requirements  (Brown,  1994). 

The  prior  SGT  effort  resulted  in  a  program  that  provided  the  initial  training  for  staff  groups, 
CPs,  and  full  staffs  (the  crawl  stage  of  training  for  staff  groups,  CPs,  and  staff)  needed  to  prepare 
for  the  less  structured  staff  exercises  delivered  in  the  Janus  or  Brigade/Battalion  Battle 
Simulation  (BBS)  simulation  environment.  The  current  program  further  refined  the  SGT  user 
tools  to  support  implementing  the  training  program  at  the  unit’s  home  station,  while  enhancing 
the  techniques  and  procedures  for  assessing  staff  performance  and  conducting  AARs. 

Scope  of  the  Project 

The  Team  conducted  a  front-end  analysis  consisting  of  a  literature  review  and  a  limited 
training  needs  assessment.  This  provided  the  developers  with  an  understanding  of  both  the 
training  needs  of  an  armor  brigade  staff  and  the  unit  staff  training  constraints  in  the  current  Army 
training  environment.  The  SIMBART  OPORDs  were  used  as  the  baseline  upon  which  the 
trigger  event-based  exercises  were  developed.  Each  complete  exercise  (preview,  preparation, 
execution  and  feedback)  was  designed  to  be  conducted  in  a  four-hour  block.  The  TSP  included 
train-the-trainer  (T^)  components,  preparation  and  execution  materials,  observer  materials,  and 
AAR  and  take  home  package  (THP)  materials.  The  current  effort  uses  the  same  computer 
platform  as  the  earlier  SGT,  with  hardware/software  modifications  to  support  the  delivery  of  T^ 
instruction. 


Organization  of  the  Report 

The  remainder  of  this  report  contains  the  following  chapters: 

Design  -  A  description  of  the  training  project  design  including  decisions  that  were  made  and 
rationales  for  those  decisions. 

Training  Support  Package  Development  -  A  discussion  of  the  process  for  developing  the 
TSPs,  including  decisions  made  and  results  of  reviews  of  the  initial  products.  This  discussion 
includes  development  of  the  unit  preparation,  tactical,  and  trainer  materials  and  the  support 
requirements  for  the  project. 
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Formative  Evaluation  -  A  description  of  the  formative  evaluation  process  and  a  summary  of 
key  findings  during  external  tests  of  the  training  project. 

Lessons  Learned  -  A  summary  of  the  most  important  lessons  learned  during  the  project. 

Conclusions  and  Recommendations  -  The  findings  and  suggestions  for  eontinued  research. 
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DESIGN 


Overview  of  the  Staff  Group  Trainer  Design  Concept 

Most  team  training  programs  do  not  attempt  to  teach  team  skills.  Instead,  they  teach 
individual  skills  to  a  team  (Blinkensderfer,  Cannon-Bowers,  &  Salas,  1994).  The  SGT  program 
was  designed  to  teach  team  skills  and  processes  to  a  military  staff. 

Figure  1  depicts  the  SGT  project  as  a  bridge  connecting  individual  training  (accomplished  in 
BSTS)  and  collective  training  (accomplished  in  structured  training  programs  such  as  SIMBART 
and  COBRAS).  Portraying  the  levels  of  the  SGT  (staff  transition,  staff  integration,  and  CP 
tables)  in  ascending  order  along  the  arch,  the  figure  demonstrates  the  stepwise  nature  of  the  team 
skills  development  process  (Brown,  1994;  Martens,  1990).  At  the  beginning  of  the  bridge — 
during  the  staff  transition  table — ^teams  focus  on  the  basics  (the  knowledge  level),  while  at  the 
mature  stage — during  the  command  post  table — ^there  is  a  melding  of  teamwork  and  taskwork 
(Bloom,  Engelhart,  Furst,  Hill,  &  Krathwohl,  1956)  such  that  these  activities  become 
indistinguishable  (McIntyre,  Morgan,  Salas  &  Glickman,  1988).  These  concepts  reflect  the 
Army’s  crawl,  walk,  and  run  operational  training  approach  (Department  of  the  Army,  1988b). 


Figure  1.  The  Force  XXI  training  bridge. 
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Ideally,  staff  officers  participating  in  the  SGT  project  have  had  BSTS  training.  This  training 
develops  an  understanding  of  individual  staff  responsibilities.  An  officer’s  staff  section  begins 
with  the  staff  transition  table.  This  table  trains  the  staff  members  to  work  together  within  their 
sections.  The  section  assesses  its  performance  using  the  SGT  built-in  assessment  tools  and  then 
proceeds  when  ready  to  the  staff  integration  tables.  These  tables  focus  on  training  the  staff  on 
the  required  techniques  and  procedures  necessary  to  integrate  their  individual  staff  section 
functions  into  those  of  other  staff  sections  within  the  CP.  As  with  the  staff  transition  table,  each 
section  assesses  its  performance;  however,  each  section  also  participates  with  the  other  sections 
in  evaluating  the  staff  integration  skills.  In  the  culminating  tables  (CP  exercises),  the  staff 
sections  all  work  together  in  supporting  the  execution  of  the  commander’s  plan  by  synchronizing 
unit  actions  in  response  to  the  commander’s  directives.  After  concluding  this  table,  the  staff 
sections  conduct  both  individual  and  group  performance  assessments.  The  commander  receives 
a  comprehensive  staff  performance  assessment  and  action  plan  which  suggests  how  to  proceed 
with  training  to  ensure  unit  success  in  other  constructive  staff  exercises. 

Theoretical  Support  for  the  Staff  Group  Trainer 

The  SGT  design  is  built  on  research  related  to  teams,  team  processes,  team  strategies, 
teamwork,  shared  mental  models,  and  adult  learning  conducted  diuing  the  last  15  years.  The 
definitions  and  constructs  of  this  literature  which  were  useful  for  this  project  are  discussed  in 
Appendix  B.  The  SGT  team  used  this  body  of  literature  to  establish  that  a  brigade  staff 
constituted  a  team.  This  permitted  the  theoretical  models,  measurement  constructs,  and  analytic 
methodologies  on  teams  which  appeared  in  published  literature  to  be  applied  to  the  current  effort. 

The  current  effort  built  on  concepts  and  approaches  discussed  by  Bailey,  Johnston,  Smith- 
Jentsch,  Gonos,  &  Cannon-Bowers  (1995).  These  researchers  provided  guidelines  for  the  design 
of  training  scenarios  with  event-based  performance  triggers,  the  development  of  team 
performance  measures,  the  design  of  a  performance  feedback  system,  and  lastly,  the  development 
of  T^  training  on  providing  feedback  which  assists  in  facilitating  the  acquisition  of  teamwork 
skills.  In  addition  to  the  Bailey  et  al.  (1995)  work.  Hall  et  al.’s  (1993)  work  on  team  tactical 
decision  making  under  stress  provided  this  project  with  a  specific  approach  to  the  design  of 
structured  scenarios.  This  effort  resulted  in  an  Army  staff  training  methodology  that 
incorporated  these  theoretical  concepts  and  led  to  the  demonstration  of  its  utility  with  a  single 
sample  brigade  staff  test  of  the  training.  The  research  focused  on  interactions  between  critical 
staff  elements  and  tied  the  observation  and  assessment  of  staff  behaviors  to  specific  learning 
objectives. 


Needs  Assessment 

This  effort  began  with  a  needs  assessment  to  identify  the  appropriateness  of  the  proposed 
training  project  for  the  selected  target  audience.  Sredl  and  Rothwell  (1987)  state 

Assessing  needs  is  the  single  most  important  step  in  designing  human  resource 
development  efforts.  The  reason  is  simple:  all  subsequent  steps  in  preparing 
instruction  stem  from  it.  If  needs  are  misidentified,  then  much  time  and  money 
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will  be  wasted  in  misdirected  efforts.  All  of  the  key  steps  in  design . . .  will  be 
futile,  (p.  5) 

Upon  the  completion  of  a  needs  analysis,  it  is  possible  to  determine  the  characteristics  of  the 
training  audience  and  what  the  trainees  should  be  able  to  do  upon  completion  of  the  block  of 
instruction.  While  the  researcher  may  establish  an  initial  list  of  training  goals,  it  is  wise  to 
approach  those  with  practical  field  experience  to  clearly  identify  the  characteristics  of  the  target 
population  (Dick  &  Carey,  1985).  Thus,  the  Team  contacted  Army  officers  currently  serving  as 
brigade  commanders,  selected  for  brigade  command  or  with  very  recent  experience  (within  one 
year  of  departure  from  command).  A  summary  of  the  interviews  is  presented  in  Table  1 . 

The  interview  portion  of  the  needs  assessment  was  conducted  by  video-  or  audio¬ 
teleconference.  The  Team  developed  a  series  of  directed  questions  based  on  lessons  learned  in 
prior  SGT  work  and  a  review  of  needs  assessment  literature  (Dick  &  Carey,  1985;  Sredl  & 
Rothwell,  1987)  to  determine  the  amount  of  time  and  manpower  the  unit  had  available  to  train  a 
brigade  staff,  the  areas  of  staff  action  requiring  the  most  sustainment  and  improvement,  and  the 
level  of  expertise  in  the  staff  support  process.  From  the  responses,  the  Team  derived  a  user- 
based  assessment  of  the  training  needed  by  brigade  staffs,  and  how  that  training  might  best  be 
delivered. 


Staff  Group  Trainer  Training  Audience 

Based  on  the  needs  assessment  findings,  the  Team  limited  development  to  exercises  for  the 
main  CP.  The  recommended  manning  of  the  main  CP  is  shown  in  Table  2.  These  individuals 
represent  the  approximate  number  of  personnel  available  in  a  section  for  one  shift  in  the  Tactical 
Operations  Center  (TOC).  This  allows  the  commander  to  focus  the  training  on  one  shift  while 
other  shifts  could  focus  on  other  duties  or  be  used  to  support  the  training. 


Training  System 

The  SGT  hardware  suite  consists  of  seven  Sun  SPARC  workstations  linked  by  a  LAN  (see 
Figure  2).  Each  workstation  consists  of  two  19-inch  color  monitors,  a  keyboard,  a  processor,  and 
a  mouse.  A  military  tactical-style  map  is  displayed  on  the  left-hand  monitor.  A  message  display 
is  on  the  right  monitor.  One  workstation  is  used  to  control  the  exercise.  When  an  exercise  is 
executed,  this  workstation  sends  pre-scripted  messages  from  the  exercise  message  database  over 
the  LAN  to  the  other  workstations.  Two  or  more  workstations  are  allocated  for  support 
personnel  role-playing  subordinate  units  or  staffs  and  higher  HQ.  The  remaining  four 
workstations  are  allocated  across  the  staff  positions — intelligence,  operations,  fire  support  (FS), 
and  engineers  (ENs).  These  workstations  function  as  the  staff  section’s  map  board,  staff  journal, 
files,  and  all  forms  of  communication  devices.  The  system  also  has  a  large  screen  monitor  that  is 
used  to  display  the  main  CP’s  situation  map  (SitMap).  Each  staff  section  in  the  main  CP  can 
update  information  to  this  SitMap. 
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Table  1 


Summary  of  Interviews  with  Commanders  on  SGT  Design 


Training  Area  and  Issues 

Training  Approach 

Computer-based  training 


Commanders’  Recommendation 


•  Develop  as  many  products  as  possible  that  can  be  employed 
by  one  person — enable  self-paced  training,  possible 
exportability,  low  overhead. 

•  Create  user  friendly  system  to  encourage/enable  maximum 
usage  and  impose  minimum  train-up. 

•  Design  system  to  provide  compatibility  with  home  station 
hardware.  Optimize  exportability;  would  like  system 
ultimately  to  be  administered  from  unit  administrative  Local 
Area  Network  (LAN)  in  the  headquarters  (HQ). 


Workstation  operator,  system  •  Enable  self-paced  training,  lower  overhead, 
administrator,  and  training 


Training  time  available 
Training  Media 

Multimedia  use  for  previews, 
AAR,  THP,  feedback  materials, 
and  methodologies 


Training  Feedback 

Commander’s  Fact  Book 
(Now  called  Staff  Training 
Profile) 


Training  Support  Material 

Alternative  to  issuing  complete 
OPORDs 


•  Limit  training  to  no  more  than  four  hours  at  any  given  time. 


•  Ensure  consistent,  structured  presentation,  training,  and 
feedback. 

•  Automate  training  exercise  demonstration  to  ensure 
consistent,  structured  presentation  and  flow  of  SGT  exercise. 

•  Streamline  TSP  structure  to  minimize  the  amount  of  print 
materials;  combine  with  redevelopment  of  training  products  as 
multimedia  or  computer-based  to  enhance  user  friendliness 
and  ease  of  use  and  reference. 

•  Revise/develop  products  to  support  enhanced 

multimedia/computer-based  AAR  and  THP  processes. _ 


•  Change  massive  master  copy  model  of  THPs  to  an  analytical 
summary  of  staff  performance  trends.  Seeking  to  create  a 
more  relevant  document  which  is  easily  used  as  a  training 
management  tool  for  commander  or  executive  officer  (XO). 


•  Lessen  required  time  for  exercise  preparation;  Providing  only 
essential  material  defuses  the  OPORDs  quality  debate  before 
it  starts. 


Doctrinal  foundation 


•  Use  CTC  trends  for  design  baseline  only. 

•  Use  Warfighter  lessons  learned  since  focus  is  on  brigade. 
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Table  2 


SGT  Brigade  Training  Audience:  Main  Command  Post 


Section 

Personnel 

Officer-in-Charge 

XO  or  Battle  Captain 

Operations  (S3)  Section 

Assistant  S3  Officer 

Assistant  S3  Non-Commissioned  Officer  (NCO) 

S3  Assistant 

Logistics  (S4)  NCO 

Intelligence  (S2)  Section 

S2  Officer 

S2  NCO 

S2  Assistant 

Fire  Support  Element 

Fire  Support  (FS)  Officer 

FS  NCO 

FS  Assistant 

Engineer  (EN)  Section 

EN  Officer 

ENNCO 

Each  section  uses  their  workstation  to  create,  display,  and  edit  various  operational  overlays 
on  the  map  display,  enabling  the  staff  section  to  maintain  the  same  type  of  map-based 
information  as  they  would  in  a  CP.  The  message  display  provides  incoming  reports  in  standard 
message  formats.  The  staff  section  can  read,  compose,  transmit,  forward,  and  annotate  messages 
going  to  and  coming  from  other  CPs  (as  emulated  from  within  the  exercise  system). 

Based  on  lessons  learned  in  previous  SGT  work,  the  Team  developed  software  to  enhance 
some  of  the  system’s  features,  increase  its  reliability,  and  make  it  easier  to  use.  These  system 
enhancements  are  described  in  Appendix  D. 
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Figure  2.  The  Staff  Group  Trainer  hardware  suite. 


Learning  Objectives 

The  learning  objectives  for  the  current  effort  benefited  fi'om  the  earlier  SGT  and  SIMBART 
efforts  which  had  already  incorporated  the  Battlefield  Function  analyses  (Ford,  Mullen,  & 

Keesling,  1997).  The  SGT  learning  objectives  are  shown  in  Table  3.  They  reflect  the 
implementation  and  complete  integration  of  learning  objectives  through  all  dimensions  of  the 
project. 
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Table  3 


Learning  Objectives  for  the  Brigade  Training  Project 


Learning  Objective  Description _ 

Monitor  unit  operations  Each  section  actively  seeks  information  about 

•  higher, 

•  adjacent, 

•  supporting,  and 

•  subordinate  units. _ 

Each  section  acquires  information  by 

•  listening  to  reports  and 

_ •  asking  for  needed  information. _ 

Process  information  and  Each  section 

messages  •  collates, 

•  transforms,  and 

•  organizes  information. _ 

Each  section  stores  information  on 

•  maps, 

•  situation  boards, 

•  journals,  and 

•  files. 

_  All  information  can  be  retrieved  and  used. _ 

Analyze/evaluate  Each  section  informally  (i.e.,  a  mental  evaluation  of  a  recurring  report)  and/or 

information  formally  (i.e.,  a  detailed  examination  of  a  specific  subject,  or  a  staff  estimate) 

attaches  meaning,  either  speculative  or  confirmed,  to  information  that  has  been 

_  acquired. _ 

Communicate  mission  Each  section  transmits  mission  critical  information  or  intelligence  to  those  who 
critical  information  must  make  decisions  about  or  act  on  it.  This  includes  initial  transmittal  of  sensed 

information;  relaying;  and  disseminating  throughout  the 

•  staff, 

•  CPs, 

•  subordinate  units, 

•  supporting  units,  and 

_ •  higher  HQ. _ 

Coordinate  information  Each  section  exchanges  and  discusses  information  and  intelligence  with  others 

and  intelligence _ outside  the  section  to  clarify  meaning  and  determine  implications. _ 

(table  continues) 
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Table  3  (Continued) 

Learning  Objectives  for  the  Brigade  Training  Project 


Learning  Objective 

Description 

Integrate  staff  input 

The  XO/Battle  Captain  aids  the  commander’s  battlefield  awareness  by: 

•  combining  information  and  intelligence  from  all  staff  sections, 

•  putting  information  and  intelligence  into  a  useable  format,  and 

•  passing  information  and  intelligence  to  the  commander. 

The  XO  or  Battle  Captain  identifies  areas  requiring  staff  sections  to  combine 
efforts  to  support  the  commander’s  intent. 

Recommend  a  course 

XO  or  Battle  Captain  and  staff  sections  develop  and  analyze  courses  of  action. 

of  action 

XO  or  Battle  Captain  recommends  a  course  of  action  (CO A)  to  the  commander. 

Disseminate 
commander’s  decision 

The  staff  prepares  and  issues  orders  or  fragmentary  orders  (FRAGOs)  to  inform 
units  and  staff  of  commander’s  decision. 

Synchronize  activities  of 
subordinate  and 
supporting  units  and 
direct  Battlefield 

Operating  System  (BOS) 
assets  to  support  the 
commander’s  intent. 

The  XO  or  Battle  Captain  in  conjunction  with  each  section  arranges  and  directs 
the  BOS  assets  to  agree  in  time  or  rate,  ensuring  their  efforts  are  aligned  to  execute 
the  commander’s  intent  or  direction.  The  XO  or  BC  and  each  section 

•  track  activities  of  BOS  assets  and 

•  intervene,  if  required,  to  ensure  their  activities  support  the  commander’s  intent. 

Exercise  Library 

Though  the  SGT  project  developed  only  a  limited  set  of  prototype  exercises,  the  Team 
conceptualized  an  integrated  training  system,  consisting  of  the  SGT  conceptual  mission  library 
shown  in  Table  4.  This  library  evolved  from  the  integration  of  known  staff  training  deficiencies 
as  identified  by  the  Center  for  Army  Lessons  Learned,  the  needs  assessment  field  interviews,  and 
related  sources.  The  training  exercises  are  divided  into  three  tables:  the  staff  transition  table 
(which  focuses  on  BOS  staff  section  actions),  the  staff  integration  table  (which  focuses  on  inter¬ 
staff  BOS  integration  skills),  and  the  command  post  table  (which  focuses  on  overall  staff 
performance).  All  the  exercises  use  the  Area  Defense  mission;  it  was  selected  to  provide  the 
contextual  background,  environment,  and  tactical  scenario  for  the  training.  A  discussion  of  the 
decision  to  use  this  mission  and  the  specific  exercises  it  contains  can  be  found  in  the  Training 
Support  Package  Development  Chapter. 

The  change  from  a  dedicated  0/C  team  and  a  reduction  in  the  number  of  training  support 
personnel  required  the  Team  to  make  drastic  changes  to  the  program.  These  modifications  were 
primarily  in  the  area  of  the  training  team  roles,  the  T^  program,  and  the  training  team  job  aids. 
This  section  will  discuss  those  changes. 
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Table  4 


SGT  Conceptual  Mission  Library 


Exercise  Title 

Exercise 

Training  Audience 

Selected  for 

# 

(Section(s)) 

prototype 

development 

Staff  Transition  Table 

Coordinate  support  to  security  force  battle 

1 

Personnel  (Sl)/S4 

No 

Execute  Reconnaissance  and  Surveillance 
(R&S)  during  security  force  battle 

2 

S2 

Yes 

Monitor  execution  of  security  force  battle 

3 

S3 

Yes 

Coordinate  &  execute  support  operations  within 
the  Brigade  Support  Area  during  security  force 
battle 

4 

Support  Operations 

No 

Execute  FS  for  the  security  force  battle 

5 

Fire  Support  Element 

Yes 

Mobility/counter  mobility/survivability  support 
to  security  force  battle 

6 

Engineer 

Yes 

Direct  brigade  staff  during  security  force  battle 

7 

Brigade  XO 

No 

Staff  Integration  Table 

Execute  R&S  Plan 

Determine  enemy  main  effort 

8 

S2,  S3,  BC 

No 

Mobility  &  survivability; 

Establish  engagement  area 

9 

S2,  S3,  BC,  FSE,  EN 

No 

Integrate  fires  &  maneuver: 

Execute  engagement 

10 

S2,  S3,  FSE,  BC 

Yes 

Force  protection: 

Coordinate  passage  of  lines 

11 

S2,  S3,  Air  Defense 
Artillery  (ADA), 
Commander,  EN,  BC 

No 

Concurrent  Planning: 

Develop  a  tentative  attack  plan  during  defensive 
operations 

12 

S3  (plans),  S4,  BC, 
XO,  FSE,  Signal 
Officer 

No 

Command  Post  Table 

Synchronize  combat  service  support  plan 

13 

REAR  CP 

No 

Alternate  Mission,  Enemy ,  Terrain,  Troops, 

Time  Available  (METT-T)  to  #13 

14 

REAR  CP 

No 

Identify  counterattack  options 

15 

MAIN  CP 

Yes 

Alternative  METT-T  to  #15 

16 

MAIN  CP 

No 

Execute  FRAGO 

17 

Tactical  CP 
(TAC  CP) 

No 

Alternate  METT-T  to  #17 

18 

TACCP 

No 

15 


Training  Team  Roles 


Table  5  lists  the  training  team  positions  and  desired  qualifications  for  each  of  those 
positions.  The  SGT  team  anticipated  that  the  key  positions  (exercise  director,  Senior  Observer, 
and  possibly  the  higher  HQ  interactor)  would  be  filled  by  the  brigade  command  structure.  The 

Table  5 

Training  Team  Positions  and  Qualifications  for  SGT 


POSITION 

RANK 

QUALIFICATION 

Exercise  Director 

Colonel  or 

Brigade  commander  or  XO, 

Lieutenant  Colonel 

Experience  in  brigade  staff  operations,  successful 

(LTC) 

XO  or  S-3,  Command  &  General  Staff  College 
(CGSC)  &  War  College  Graduate 

Armor  (AR)  or  Infantry  (IN)  branch 

Senior  Observer 

LTC 

Experienced  in  brigade  staff  operations 

CGSC  graduate 

AR  or  IN  branch 

Staff  Observer 

Major  (MAJ)  or 

Experienced  in  brigade  staff  operations 

Captain 

CGSC  or  Combined  Arms  Staff  Services  School 
(CASQ  graduate  (CGSC  preferred) 

or 

AR,  IN,  EN,  or  Field  Artillery  (FA)  branch 

Experienced  in  brigade  staff  operations 

SGM/Master 

SGM  Academy  graduate  preferred 

Sergeant  (MSG) 

AR,  IN,  EN,  or  FA  branch 

Interactor  (higher  HQ) 

MAJ 

Division  or  brigade  staff  operations  experience 

or 

CGSC  or  CAS^  graduate  (CGSC  preferred) 

Division  or  brigade  staff  operations  experience 

SGM/MSG 

Combat  arms  experience 

SGM  Academy  graduate  preferred 

Interactor 

MAJ 

Brigade  combat  arms  or  battalion  staff  operations 

(Subordinate/Adjacent 

experience 

HQ) 

or 

CAS^  graduate 

Brigade  combat  arms  or  battalion  staff  operations 

MSG/Sergeant 

experience 

First  Class 

Advanced  NCO  Course/Battle  Staff  NCO  Course 
graduate 

Exercise  Administrator 

Officer 

Basic  knowledge  of  Windows  95®  and  Microsoft 

(May  be  one  of  the 

Or 

Office® 

Interactors) 

NCO 
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staff  observer  and  the  other  interactor(s)  could  be  members  of  the  brigade  staff  who  were  not 
being  trained  during  the  exercise.  The  exercises  were  designed  for  the  commander,  XO,  S3, 
command  sergeant  major  (SGM),  and  operations  sergeant  major  to  be  key  people  in  conducting 
the  training.  Involvement  of  the  commander  and  XO  were  essential  to  the  training  project. 

These  two  people  must  train  their  staff  to  meet  their  own  informational  needs.  Without  their 
involvement,  the  project  would  not  have  the  needed  direction  and  feedback. 

Train-the-Trainer  Program 

The  change  from  a  dedicated  training  team  at  Fort  Knox  to  a  training  team  using  brigade 
assets  changed  the  requirements  for  the  T^  program.  The  T^  program  would  need  to  meet  the 
requirements  of  less  train  up  time,  fewer  support  personnel,  and  less  knowledge  about  the  SGT 
training  system.  This  combination  of  constraints  was  far  different  (and  much  more  demanding) 
than  for  previously  developed  T^  programs.  The  involvement  of  key  members  of  the  brigade 
command  structure  as  trainers  meant  that  the  T^  program  would  have  to  be  relatively  short, 
preferably  less  than  four  hours.  It  would  have  to  provide  an  overview  of  the  program  to  include 
the  conceptual  basis  of  the  program.  Because  the  training  team  structure  would  be  unknown  and 
possibly  come  with  a  diverse  skill  set,  the  Team  anticipated  the  need  for  multiple  job  aids  to 
assist  the  training  team  in  conducting  the  training.  The  T^  program  would  have  to  focus  on 
familiarizing  the  team  with  the  job  aids  and  their  use  in  the  training  program.  Since  the 
scheduling  of  the  staff  training  might  be  spread  over  several  weeks  or  even  months,  some  form 
of  refresher  training  and  a  section  focused  on  each  exercise  would  have  to  be  provided. 
Preferably  these  packages  should  be  designed  for  computer-based  implementation  to  ensure  that 
information  could  be  delivered  consistently  and  would  not  be  lost  or  misplaced. 

Structured  Scenario  Design 

Numerous  investigators  (Bandura,  1986;  Britton  &  Tesser,  1982;  Manz  &  Sims,  1981; 
Norris,  1986;  Smith,  1994;  Thornton  &  Cleveland,  1990)  have  indicated  that  placing  unprepared 
individuals  in  a  simulation  training  environment  before  their  basic  skills  are  developed  may 
cause  the  trainee  to  miss  underlying  relationships  and  learning  objectives.  To  prevent  this, 
highly  structured  scenarios  should  be  specifically  designed  to  elicit  appropriate  decision  making 
actions  or  critical  team  actions  (Hall  et  al.,  1993).  The  SGT  team  used  a  systematic  procedure  to 
design  scenarios  with  adequate  structure  to  focus  on  facilitating  the  team  process  learning 
objectives. 

Training  Team  Job  Aids 

Because  of  the  structure  of  the  training  project  and  the  anticipated  inexperience  of  the  unit 
training  team,  the  SGT  team  determined  that  there  was  a  requirement  for  job  aids  for  most 
training  team  members.  To  get  the  training  started,  the  exercise  administrator’s  instructions 
would  need  to  allow  for  easy  initiation  of  the  system  and  the  exercise.  The  preparation  material 
would  have  to  be  scripted  and  provide  simple  instructions  for  use  by  the  training  team  to 
immerse  the  staff  into  the  exercise  in  a  short  time.  The  job  aid  for  the  observers  would  contain 
prompts  on  when,  where,  and  what  to  look  for  in  terms  of  specific  staff  actions  in  response  to  the 
system  delivered  message  cues.  The  interactor  cell  would  be  provided  job  aids  with  very  specific 
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guidelines  on  how  to  respond  to  requests  for  information  (RFIs)  from  the  staff  sections.  The 
information  to  answer  these  inquiries  would  be  easily  retrieved  during  the  conduct  of  the 
exercise.  The  section  and  exercise  AARs  would  be  easy  to  follow  and  focused  on  specific 
learning  objectives  for  each  exercise.  The  following  sections  discuss  the  exercise  segments  and 
some  of  the  ways  the  design  addressed  the  needs  of  the  training. 

Exercise  Segments 

The  SGT  project  team  organized  the  exercises  into  segments  to  facilitate  exercise 
development  and  administrative  control  during  execution.  These  segments  began  when  a  unit 
was  at  the  site  and  had  made  its  preparations  for  conducting  the  training  program.  Each  exercise 
was  broken  into  three  segments:  (a)  exercise  preparation  (actions  taken  prior  to  the  start  of  the 
exercise  which  provide  background  and  final  coordination  opportunities),  (b)  exercise  execution 
(actual  running  of  the  simulation  from  start  to  finish),  and  (c)  exercise  feedback  (AARs  and  self- 
assessment  activities). 

The  following  subsections  discuss  the  design  of  these  three  segments.  The  methods  for 
developing  the  indicated  materials  are  discussed  in  the  Training  Support  Package  Development 
Chapter. 


Exercise  Preparation 

The  SGT  team’s  design  aligned  closely  with  BroAvn’s  (1992)  approach.  Therefore,  the  SGT 
design  was  based  on  the  premise  that  the  staff  could  receive  a  thorough  grounding  on  the  specific 
and  general  situation  for  the  exercise  and  become  immersed  in  the  battle  context  in  30  minutes  or 
less.  The  Team  divided  the  exercise  preparation  into  three  parts:  exercise  preview,  section 
preparation,  and  pre-execution  staff  huddle.  The  following  sections  explain  the  SGT  design  for 
these  three  parts. 

Exercise  Preview 

The  exercise  preview  aligned  closely  with  Brown’s  (1992)  general  situation.  The  SGT  team 
envisioned  the  general  situation  summary  to  be  a  short  battle  summary  to  the  entire  staff, 
consisting  of:  (a)  what  had  led  up  to  this  point  in  the  battle,  (b)  what  to  expect  in  the  exercise, 
and  (c)  a  description  of  the  exercise  learning  objectives. 

Based  on  the  success  of  the  multimedia  preview  created  previously,  the  design  concept  was 
that  this  should  be  primarily  a  multimedia  presentation.  However,  the  designers  also  wanted  this 
multimedia  preview  to  provide  an  opportunity  for  the  commander  to  state  his  expectations  for  the 
staff  and  to  emphasize  the  learning  objectives  on  which  he  wanted  the  staff  to  focus.  They 
expected  the  commander  to  focus  on  his  expectations  concerning  critical  pieces  of  information  he 
needed  (Commanders  Critical  Information  Requirements  [CCIR])  or  a  recommendation  based  on 
the  staff  s  analysis  of  the  situation  that  he  would  expect.  The  previous  SGT  team  had  allowed 
the  commander  to  present  this  portion  of  the  preview,  but  had  not  helped  the  commander  to  focus 
on  the  exercise  learning  objectives.  Additionally,  the  SGT  was  designed  to  be  more  tum-key. 

As  a  result,  the  Team  decided  this  portion  of  the  preview  would  also  be  presented  in  the 
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multimedia  format  but  would  then  allow  the  commander  the  opportunity  to  highlight  certain 
aspects  or  explain  how  he  wanted  the  information  presented  to  him.  The  concept  allowed  the 
Team  to  ensure  the  information  was  presented.  At  the  same  time,  it  provided  the  commander  an 
opportunity  to  personalize  the  presentation.  The  intent  of  the  preview  was  to  stress  the  plan  and 
the  shared  situation  model  or  shared  mental  model.  The  focus  on  the  CCIR,  Decision  Support 
Template  (DST),  and  synchronization  matrix  highlighted  the  shared  model  and  the  staffs  roles 
and  responsibilities. 

During  the  preview,  the  staff  also  would  converge  their  action  plans  from  a  previous 
exercise  with  the  specific  situations  presented  in  the  preview.  Specifically,  this  part  would 
include  a  review  of  the  action  plan  prepared  in  an  earlier  exercise  with  a  look  at  how  this  plan 
would  be  applied  in  this  exercise.  This  review  would  not  be  conducted  if  this  was  the  first 
exercise  the  staff  was  conducting.  Either  the  commander,  XO,  or  battle  captain  (BC)  would 
conduct  this  part  of  the  exercise.  The  designers  allotted  approximately  10  to  15  minutes  for  the 
entire  exercise  preview. 

Staff  Section  Preparation 

The  staff  section  preparation  component  was  designed  to  get  each  individual  staff  officer 
and  small  staff  team  fully  immersed  into  the  exercise  as  conceived  by  Brown’s  (1992)  “specific 
situation”  notion.  Earlier  SGT  lessons  learned  had  demonstrated  that  this  would  be  difficult. 
Lack  of  a  dedicated  0/C  team  and  the  need  to  adequately  prepare  the  unit-provided  training 
support  personnel  made  the  simplification  of  the  preparation  training  requirements  a  necessity. 
As  a  result,  the  designers  would  need  to  make  the  preparation  much  easier  to  implement. 

Structured  training  often  includes  the  use  of  a  previously  prepared  OPORD  for  the 
exercise(s).  The  staff  executing  the  SGT  OPORD  was  not  involved  in  the  preparation  of  the 
OPORD  and  the  analysis  that  went  into  the  development  of  the  OPORD.  Part  of  the  staff 
preparation  must  take  the  staff  quickly  through  specific  parts  of  the  OPORD  to  better  imderstand 
why  certain  decisions  were  made.  The  Team  decided  that  this  in-depth  preparation  would  best 
be  conducted  by  each  section  individually,  with  the  section  personnel  examining  the  tools  they 
would  have  developed  (e.g.,  synchronization  matrix,  DST,  power  charts,  CCIR,  priority 
intelligence  requirements  and  which  they  will  need  to  use  during  the  exercise).  This  was  done 
using  self-contained  Visual  Basic®  programs  customized  to  the  needs  of  each  staff  section  for  a 
given  exercise. 

The  designers  allotted  approximately  30  minutes  for  staff  section  preparation.  In  the  pilots 
and  the  trials,  this  amount  of  time  was  only  needed  for  the  first  iteration.  Subsequent  iterations 
took  only  10-15  minutes. 

Pre-Execution  Staff  Huddle 


The  third  stage,  pre-execution  staff  huddle,  did  not  come  from  Brown  (1992).  Rather,  it 
evolved  from  the  research  on  shared  mental  models  and  effective  team  leader  practices  (Orasanu, 
1990;  Oreisanu  &  Salas,  1993).  This  phase  allows  the  XO  or  BC  to  gather  his  principal  staff 
officers  within  the  main  CP  and  ensure  they  understand  his  and  the  commander’s  expectations 
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for  the  upcoming  exercise.  As  a  result  of  this  pre-execution  staff  huddle,  each  staff  officer 
should  come  to  share  the  mental  model  of  the  commander  and  XO,  and  understand  his  role  and 
expected  contribution. 


Exercise  Execution 


Level  of  Difficulty 

The  designed  exercises  needed  to  be  challenging  while  providing  a  high  likelihood  of 
success.  The  challenge  of  the  exercises  must  match  the  ability  of  the  participants.  This  is 
referred  to  as  the  “flow”  experience  (Martens,  1990,  p.  45).  When  the  challenge  of  a  task  exceeds 
the  ability  of  the  participants,  the  participants  become  anxious  or  frustrated.  When  the  challenge 
is  below  the  participants’  ability,  they  become  bored.  However,  when  the  challenge  of  the  task 
matches  the  ability  of  the  participants,  the  participants  become  caught  up  in  or  “immersed”  in  the 
activity.  According  to  Martens  this  is  when  the  “flow”  state  is  achieved.  Berliner  (1985)  found 
that  participants  do  not  have  to  be  successful  all  the  time  to  be  in  this  flow  state,  but  have  to 
believe  they  have  a  high  likelihood  of  success  (successful  about  75  to  80%  of  the  time).  Higher 
success  rates  may  lead  some  participants  to  feel  the  exercise  is  not  challenging  enough. 

Achieving  this  flow  state  in  training  is  complicated  because  staffs  and  staff  sections  are 
often  not  at  the  same  level  of  training.  A  large  set  of  seamlessly  integrated  exercises  is  required 
to  achieve  the  flow  needed  for  a  robust  and  varied  training  environment.  This  program  provides 
a  discrete  subset  of  these  exercises  needed  to  prove  the  concept. 

System  or  Interactor  Created  Cues 

The  SGT  messages  are  designed  to  provide  the  cues  (e.g.,  spot  reports  from  subordinate 
units;  intelligence  reports  from  higher  HQ)  to  the  staff  members  which  will  elicit  the  desired 
behaviors  that  achieve  the  learning  objectives.  The  critical  messages  act  as  the  superstructure 
within  the  exercises  that  keep  the  participant  within  the  “flow”  window.  These  messages 
represent  realistic  (tactically  correct  and  doctrinally  sound)  events,  and  are  appropriately  timed  to 
keep  the  exercise  at  the  correct  pace.  Through  the  table-to-table  progression,  cues  elicit  more 
complex  staff  processes,  thus  increasing  the  level  of  difficulty.  This  was  intentional,  bracketing 
the  exercise  within  the  challenge/ability  flow  while  maintaining  the  crawl-walk-run  training 
philosophy.  This  heavy  reliance  on  the  system  to  provide  cues  simplifies  the  training  team 
requirements. 

Coaching 

There  is  a  significant  need  for  coaching  to  provide  clues  to  the  staff  on  their  expected 
performance  during  execution.  Especially  in  the  early  tables,  coaching  fiimishes  reminders  to  the 
staff  on  what  they  should  be  doing  throughout  the  exercise.  In  SGT,  coaching  is  provided  by 
observers,  using  questions  and  checklists  automatically  cued  up  on  personal  digital  assistants 
(PDAs).  Tied  to  the  events,  these  questions  are  specifically  linked  to  the  actions  of  each 
staffistaff  member,  based  on  the  learning  objectives  and  the  mission/tactical  situation.  They 
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provide  important  hints  to  spur  the  appropriate  staff  activity  and  keep  the  staff  on  track.  This 
system  simplifies  the  observers’  tasks,  making  train-up  and  execution  much  less  time  consuming. 

Exercise  Feedback 


General  Design  Concept 

The  Team  designed  the  AARs  and  trainer  instructions  so  that  feedback  to  the  staff  would 
focus  on  the  performance  of  staff  actions.  The  feedback  goal  was  to  praise  good  performance 
and  avoid  harsh  criticism  of  poor  performance.  This  feedback  design  supported  building  and 
maintaining  high  self-esteem  in  the  participants.  Bandura  (1986)  found  that  expected  outcomes 
depended  heavily  on  a  person’s  self-confidence  in  their  ability  to  perform  a  skill.  The  SGT 
feedback  design  also  supported  the  adult  learning  concept  that  to  maximize  learning, 
instructional  strategies  should  provide  a  climate  that  is  supportive  and  that  avoids  unpleasant 
experiences  or  threats  to  self-esteem  (Laird,  1985;  Marshak,  1983;  Sredl  &  Rothwell,  1987; 
Wlodkowski,  1985a,  1985b;  Zemke  &  Zemke,  1981).  This  was  done  using  Johnson  and 
Johnson’s  (1994)  suggestions  that  feedback  should  focus  on  how  team  members  are  functioning 
as  a  team.  This  also  meant  that  the  team  should  examine  their  actions  and  decide  what  they 
wanted  to  sustain  or  improve. 

Section  After  Action  Review 

Commanders  indicated  that  the  number  of  support  people  required  for  the  training  should  be 
reduced  to  four  or  five.  To  achieve  this,  the  SGT  team  eliminated  some  of  the  section  observers 
and  expanded  the  role  of  the  training  participants.  This  meant  that  the  Team  had  to  look  for 
different  methods  to  conduct  section  AARs.  Blickensderfer,  Cannon-Bowers,  and  Salas  (1994) 
proposed  four  techniques  to  foster  and  train  self-correction.  These  four  techniques  were:  (a) 
provide  time  for  team  discussions,  (b)  model  and  role  play  feedback,  (c)  use  a  facilitator  to 
record  important  incidents  or  specific  times  when  performance  faltered,  and  (d)  have  each  team 
member  complete  ratings  of  the  team’s  performance  on  various  dimensions  and  then  discuss  and 
compare  the  ratings.  The  SGT  team  adopted  these  techniques  (except  for  model  and  roleplay)  as 
part  of  the  section  AAR  process.  This  was  implemented  using  a  Visual  Basic®  program  that 
leads  each  section  through  an  individualized  exercise  review. 

The  individualized  exercise  review  program  was  designed  and  developed  using  techniques 
identified  by  Smith- Jentsch,  Payne,  &  Johnston  (1996).  In  conducting  research  in  guided  team 
self-correction,  a  guide  was  developed  which  “summarizes  each  event  (what  actually  happened), 
provides  open-ended  questions  regarding  targeted  behaviors  (positive  and  negative)  organized  by 
teamwork  skill,  and  provides  a  format  for  setting  goats  imder  each  teamwork  skill”  (p.  1 8).  They 
found  that  this  method  enhanced  the  learning  of  the  teams  that  used  it  in  an  actual  shipboard 
setting.  The  Team  adapted  this  method  of  exercise  review  to  the  SGT  training  environment  to 
ensure  that  the  section  AARs  followed  these  steps. 
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Exercise  After  Action  Review 


The  commanders  indicated  that  they  wanted  a  more  focused  AAR  than  they  were  receiving 
at  the  CTCs.  They  agreed  with  the  concepts  provided  by  the  Team  during  their  discussions;  these 
concepts  corresponded  with  the  feedback  guidelines  in  Bailey  et  al.  (1995).  The  Team  decided  to 
use  the  recommendations  for  the  exercise  AAR  debriefing  guide  (discussed  earlier  in  this 
chapter)  to  make  a  multimedia  presentation  and  guide  for  use  by  the  senior  observer  and 
commander  when  acting  as  a  training  facilitator  during  the  AAR. 

Commander’s  Staff  Training  Profile 

The  user  panel  commanders  indicated  they  needed  a  THP  that  was  concise  and  focused. 

With  this  in  mind,  the  Team  designed  a  package  which  contained  information  gathered  from  the 
computer  instrumentation,  the  section  AAR,  the  exercise  AAR,  and  the  observers.  This  was 
provided  to  the  commander  before  leaving  the  training  site.  The  information  consisted  of:  (a)  a 
record  of  the  commander’s  training  goals,  (b)  the  commander’s  selection  of  criteria  for  the 
exercise  AAR,  (c)  a  copy  of  data  and  user  input  from  the  section  AAR  (to  include  the  self- 
assessments),  (d)  a  trends  analysis  of  staff  performance  from  the  instrumentation  (primarily, 
message  handling  and  actions  taken  over  the  course  of  the  exercise),  (e)  a  summary  of  the 
findings  from  the  exercise  AAR,  and  (f)  a  staff-prepared  Action  Plan  from  the  exercise  AAR. 

The  important  outcomes  of  the  design  were  the  operationalization  of  theoretical  principles 
into  specific,  useable,  training  feedback  products.  The  most  important  of  these  were  the  window 
of  opportunity  (WOO)  and  the  staff  fingerprint.  These  products  provide  the  basis  for  the  actions 
taken  by  the  staff  members  in  the  conduct  of  their  activities.  A  discussion  of  these  products  and 
their  place  within  the  SGT  training  system  is  in  the  next  chapter. 
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TRAINING  SUPPORT  PACKAGE  DEVELOPMENT 


Structural  Organization 
Packaging  Style 

The  SGT  TSP  contained  paper-based,  computer-based,  and  multimedia  products  (i.e.,  T^, 
workstation  operator  training  (WST),  message  databases,  previews  and  AAR  materials).  Table  6 
shows  the  structural  organization  and  delivery  format  of  the  SGT  TSP. 

Table  6 

Structural  Organization  of  the  SGT  TSP 


TSP  Component 

Delivery  Format 

Program  Orientation  Guide 

paper-based 

Brigade  Training  Guide 

paper-based 

Brigade  Tables  with  End  of  Exercise  (ENDEX)  Power  Charts 

paper-based 

Workstation  Operator  Training 

Electronic 

Train-the-Trainer  Module 

electronic  and  paper-based 

Brigade  Tactical  Materials 

electronic 

The  SGT  TSP  was  organized  to  enhance  user  friendliness  for  the  trainers  and  training  staff, 
reduce  overhead  costs,  and  foster  maximum  automation.  Tactical  materials  were  moved  to  an 
electronic  format  (MSWord®)  and  scanned  bitmap  images.  Fewer  printed  training  materials  and 
training  tools  were  required  due  to  SGT’s  use  of  electronic  products  (e.g.,  observer  checklists  and 
tactical  materials).  In  addition,  since  only  one  exercise  per  table  was  fully  developed,  there  was 
no  need  to  produce  a  separate  hardcopy  volume  for  each  table.  The  training  unit  has  the  option 
to  print  the  tactical  materials  from  the  electronic  format  as  required. 

Writing  Style 

Structured  writing  makes  TSPs  easier  to  use  than  the  more  traditional,  prose-style  approach 
(Graves  &  Myers,  1997;  Koger  et  al.,  1996)  and  is  mandated  by  TRADOC  (Department  of  the 
Army,  1995a).  Consequently,  structured  writing  has  been  used  in  preparing  the  paper-based 
components  of  all  SGT  TSPs.  This  writing  style  emphasizes  (a)  separating  information  into 
small  units  by  purpose  and  function  so  it  may  be  easily  read  and  understood,  (b)  sequencing 
information  based  on  its  use  and  need,  (e)  labeling  topics  for  easy  scanning,  and  (d)  presenting 
infonnation  in  modular  units  so  it  can  be  easily  modified  (Horn,  1995). 

Innovative  Approaches  to  Solve  System  Training  Issues 

The  results  of  the  needs  assessment  established  the  user’s  requirements  and  standards  within 
the  framework  of  the  current  effort’s  manpower,  time,  and  equipment  constraints.  It  was  the  need 
to  create  a  staff  task  training  system  that  shaped  the  requirement  for  new  technology  uses.  The 
innovations  taken  by  the  Team  to  solve  these  problems  are  shown  in  Table  7.  Each  item  includes 
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a  brief  description  of  the  technology,  the  reason  or  source  for  its  inclusion  on  the  list,  and  a  brief 
discussion  of  the  implementation  approach. 

Table  7 

Innovative  Approaches  Incorporated  in  the  SGT  Training  Program 


Training  Reason/Needs 

Implementation  Approach 

•  Ease  of  operation 

Simplified  exercise  administrator’s 

• 

Menu  driven  Graphical  User’s  Interface  to  make  operation 

job 

easier,  similar  to  a  Windows  environment. 

• 

Avoid  use  of  UNIX  editor. 

Simplified  workstation  operations 

• 

Map  edit  functions  streamlined/simplified  so  icon  placement 
and  other  functions  are  faster/easier. 

• 

Map  Display  automated  to  update  actions. 

• 

Map  Edit  Screen  simplified  to  reduce  time  it  takes  to  complete 
task. 

• 

Unnecessary  functions  removed. 

Simplified  interactor  interface 

• 

Voice  recognition*  and  touch  screen  for  interactor  workstations 
to  enable  interactors  to  more  efficiently  answer  RFIs. 

Hard  copy  materials  reproduction 

• 

On  site  printing  of  TSP  training  tools  for  specific  staff  sections. 

•  Facilitation  of  training 

Graphic  feedback  approaches 

• 

WOO  to  emphasize  the  learning  objective — communicate — to 
document  activities  taken  by  staff  at  that  workstation  regarding 
the  key  message,  including  reaction  time  to  get  information  to 
commander. 

• 

Staff  fingerprint  graphic  to  show  how  key  messages  were 
handled  for  selected  learning  objectives  (monitor,  process, 
analyze,  communicate,  and  coordinate). 

THP  usability 

• 

THP  converted  to  “fact  book”  to  get  away  from  massive  CTC 
model.  (20  pages  or  less  in  length). 

• 

Tabular/graphic  document  design  to  show  user  performance 
trends  and  link  to  Mission  Training  Plan  (MTP)/Tactics, 
Techniques,  and  Procedures  (TTP)/program  learning  objectives. 

tutorial 

• 

Computer-based  instructions  for  operating  workstations  and 
implementing  the  training  program,  using  MS  Help®. 

Large  group  exercise  multimedia 

• 

Computer-based  previews  provide  training  staff  with  overview 

previews 

of  battle  situation  (places  them  within  tactical  context). 

(table  continues) 
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Table  7  (Continued) 

Innovative  Approaches  Incorporated  in  the  SGT  Training  Program 


Training  Reasons/Needs 

Implementation  Approach 

Staff  section  preparation  materials 

• 

Computer-based  presentation  of  information  requirements 
critical  to  successful  staff  section  performance. 

•  Training  flexibility 

Restrictive  terrain  database* 

• 

Design  and  development  of  exercises  sited  restrictive  terrain 
(e.g,  Korea)  thus  to  vary  METT-T  during  execution. 

•  Enhanced  turn-key  operation 

Automated  section  AAR 

• 

Automation  of  section  AAR  at  each  workstation  to  facilitate 
self  assessment  of  performance. 

Automated  observer’s  checklist 

• 

Small  hand-held  computer  to  automate  data  dump  from  0/Cs 
and  for  unit  self-assessment  ratings  and  self-generated  action 
plans. 

Audio-visual  orientation* 

• 

Introduction  to  provide  trainers  and  training  staff  short  video¬ 
tape  orientation . 

•  Face  validity 

Fax  emulation 

• 

Use  of  printers  for  long  messages  that  are  normally  delivered 
via  fax  machine  in  field  CP. 

Army  Tactical  Command  and 
Control  System  (ATCCS) 
emulation* 

• 

Development  of  ATCCS  ‘look  and  feel’. 

♦  These  recommendations  were  investigated  but  not  implemented  during  this  project.  Given  the  cost  and  technical 
difficulty,  they  were  not  deemed  critical  to  the  demonstration  of  the  small  group  (staff)  training  approach. 


Exercise  Development  Methodology 

The  SGT  TSP  development  process  used  a  codified  exercise  development  methodology 
modified  from  the  structured,  simulation-based  training  methodology  (Campbell  et  al.,  1995). 
The  Team  split  the  TSP  development  into  two  phases:  exercise  feedback  materials  development 
and  exercise  materials  development.  This  was  done  because  of  the  immense  volume  of  exercise 
material  which  needed  to  be  developed  and  front-loaded  into  the  SGT  computer  system  (i.e., 
message  traffic  and  RFI  guidelines).  This  approach  allowed  for  a  more  concentrated  emphasis 
on  integrating  each  step  of  the  development  process  with  the  project’s  learning  objectives  (see 
Figure  3). 

The  subsections  that  follow  are  keyed  by  number  to  the  blocks  in  Figure  3.  While  the 
diagram  suggests  a  sequential  execution,  the  steps  often  overlapped.  In  addition,  the 
development  of  exercise  materials  and  the  development  of  feedback  materials  occurred 
concurrently  (especially  in  the  latter  stages  of  the  project),  as  the  results  of  material  development 
directly  provided  input  to  the  AAR  development  process. 
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Mission  Development 


Determine  Training  Audience  and  Training  Program  Learning  Objectives  [1] 

The  methods  for  determination  of  the  training  audience  and  development  of  the  learning 
objectives  for  this  project  were  discussed  in  the  Design  Chapter  of  this  report. 

Design/Develop  Worksheets  f2] 

The  Team  employed  a  systematic,  structured  methodology  to  standardize  production.  With 
the  guidance  of  the  instructional  systems  design  (ISD)  specialist  and  use  of  structured  writing,  a 
series  of  development  worksheets  were  created  for  the  exercise  authors.  These  worksheets 
helped  the  authors  capture  the  components  necessary  for  consistent  and  quality  exercise 
development  and  maintain  an  organized  record  of  progress.  Worksheets  for  each  of  the 
following  topics  were  developed: 

1 .  Exercise  Preparation  (including  learning  objectives,  tactical  scenario,  and  assessment 
plan) 

2.  Required  Print  Training  Materials 

3.  Exercise  Preview 

4.  Instructions  to  Multimedia  Developer  for  Exercise  Preview 

5.  Staff  Section  Preparation 

6.  Feedback  Strategy 

7.  Key  Messages  and  Required  Staff  Actions 

8.  WOO 

9.  Overlay  Comparison 

10.  Power  Chart  Comparison 

1 1 .  Staff  Section  Self-assessment 

12.  MTP/BSTS  Remediation 

13.  StaffSupport  Process  Module 

14.  Observer  Checklist 

15.  Instructions  to  Software  Developer  for  Staff  Section  AAR 

16.  Instructions  to  Multimedia  Developer  for  Exercise  AAR 

17.  Exercise  AAR  (including  commander’s  Comments,  Techniques  and  Procedures  for 
Exercise  AAR,  Sustain/Improve  Menu  for  AAR,  and  Learning  Objectives  Self- 
assessment  for  Exercise  AAR) 

18.  Pre-execution  Staff  Huddle  Guidelines 

Develop  Tactical  Scenario  [3] 

The  Team  selected  the  area  defense  as  the  sole  tactical  scenario  for  the  exercises.  This 
mission  was  identified  as  being  the  most  challenging  in  both  preparation  and  execution,  thus 
providing  a  complex  set  of  circumstances  which  would  enhance  training  for  all  staff  members. 

In  addition,  the  area  defense  provides  unique  opportunities  to  attend  to  preparatory  activities  by 
all  the  individual  staff  members,  as  well  as  integrating  actions  between  staff  sections.  There 
were  also  clearly  defined  parts  which  would  lend  themselves  to  partitioning  for  individual 
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exercises.  These  exercises,  designed  as  individual  vignettes  under  the  umbrella  of  a  common 
tactical  scenario,  best  support  learning  objectives  tied  to  the  critical,  tactical  events  within  each 
exercise. 

Determine  Table  Learning  Objectives  [4a1 

Training  a  staff  section  in  the  nine  identified  staff  functions  necessitated  a  progressive 
“building  block”  approach  suggested  by  Brown  at  the  SIMUTA  proof-of-concept  test  (F.  Brown 
personal  commimication,  September  14,  1994).  The  SGT  program  is  divided  into  three  distinct 
tables,  each  involving  an  increasing  level  of  difficulty  (staff  transition,  staff  integration,  and  CP 
tables).  The  learning  objectives  of  the  staff  transition  table  (monitor,  process,  analyze,  and 
communicate)  did  not  depend  on  more  than  limited  interaction  with  other  staff  sections.  This 
aspect  of  the  project  allowed  staff  sections  to  be  trained  independently,  yet  concurrently,  using 
the  same  computer-generated  message  list.  The  next  set  of  exercises,  the  staff  integration  table, 
involved  training  selected,  integrated  staff  sections  (e.g.,  a  targeting  cell  made  up  of  the  XO,  S2 
and  fire  support  officer)  to  collectively  perform  increasingly  difficult  learning  objectives 
(analyze,  communicate,  coordinate,  and  integrate).  Finally,  the  command  post  table  trained  all 
staff  sections  of  the  Main  CP  to  perform  higher  level  staff  tasks  (analyze,  communicate, 
coordinate,  integrate,  recommend,  disseminate,  and  synchronize),  focusing  on  helping  the  full 
staff  develop  synergy  in  its  operations. 


Exercise  Definition 


Determine  Exercise  Learning  Objectives  Kb] 

The  content  and  time-frame  of  each  exercise  was  determined  by  selecting  the  events  that 
would  best  elicit  the  performance  of  learning  objectives  from  the  training  audience,  be  it 
individual  staff  section  or  integrated  staff  sections  within  a  CP.  While  all  tactical  events  need  not 
be  played  out  in  order  to  ensure  the  desired  staff  performance  within  any  given  exercise  or  table, 
a  chronological  progression  through  the  area  defense  mission  offered  greater  continuity  in 
learning.  The  Team  produced  a  multimedia  exercise  preview  for  each  exercise  selected  for 
development  to  reaffirm  the  training  audience’s  understanding  of  the  mission  and  reacquaint 
them  with  the  learning  objectives  for  each  exercise. 

Select  Mission  Segments  for  Exercises  fS] 

One  of  the  most  time  consuming  and  critical  development  tasks  is  to  select  the  segments  of 
the  tactical  scenario  which  best  support  the  learning  objectives.  Exercise  authors  examined  the 
tactical  scenario,  conducted  a  task  analysis  of  crucial  points  in  the  battle,  and  determined  the 
specific  times  in  the  scenario  when  events  could  be  built  into  the  play  to  trigger  critical  learning 
objectives.  It  was  important  to  keep  the  battle  flow  continuous  within  each  exercise. 

The  Team  designed  the  staff  transition  table  to  train  only  the  first  four  learning  objectives. 
The  exercises  chosen  focused  on  a  period  in  the  tactical  scenario  without  heavy  enemy  contact: 
the  counter-reconnaissance/security  force  battle.  Being  relatively  short  and  simple,  these 
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exercises  allowed  the  staff  to  master  the  initial  objectives  and  get  accustomed  to  the  training 
system. 

The  next  exercise  set,  the  staff  integration  table,  focused  on  the  brigade’s  contact  with  the 
enemy  first-echelon  forces.  This  tactical  scenario  segment  was  chosen  because  it  required  the 
staff  to  begin  interaction  between  sections,  as  well  as  with  higher  and  adjacent  units. 

The  culminating  exercise  set,  the  command  post  table,  focused  on  identifying  the 
commander’s  options  for  a  counterattack  in  the  main  battle  area.  This  tactical  segment  represents 
an  intense  period  of  enemy  contact.  It  required  the  staff  to  become  fully  interactive  between 
sections,  as  well  as  with  the  BC,  and  the  adjacent,  subordinate  and  higher  units.  Most 
importantly,  it  provided  input  to  the  brigade  commander  to  support  his  combat  decision-making 
cycle. 

Develop  Unit  Preparation  Materials  [61 

The  SGT  unit  preparation  materials  consist  of  the  Orientation  Guide,  Training  Guide, 
Brigade  Tables,  Workstation  Operator  Training  (parts  one  and  two),  T^  Program  of  Instruction 
(POI),  and  Brigade  Tactical  Materials. 

The  Orientation  Guide  provides  a  complete  overview  of  the  SGT  project  for  interested 
brigade  commanders  who  would  potentially  choose  the  project  as  a  training  option  for  their  staff 
It  summarizes  the  training  goals  and  provides  a  general  description  of  the  SGT  training  program, 
and  includes  information  on  training  strategy,  hardware,  and  the  TSP. 

The  Training  Guide  is  intended  for  use  by  trainers  in  a  unit  committed  to  using  SGT.  It 
expands  the  Orientation  Guide  by  more  specifically  describing  the  training  audience,  training 
goals,  learning  objectives,  tables,  training  strategy,  and  trainer  requirements. 

The  Brigade  Table  book  describes  the  structure  of  brigade  staff  group  training,  brigade  SGT 
trainer  tables,  and  the  exercises  within  them.  The  book  is  divided  into  four  parts;  the  training 
program  structure,  the  staff  transition  table,  the  staff  integration  table,  and  the  command  post 
table.  This  book  contains  exercise-specific  information  for  administering  and  conducting  brigade 
exercises. 

The  T^  POI  was  developed  to  instruct  personnel  assigned  as  trainers  to  effectively  administer 
and  facilitate  brigade  exercises.  The  training  audience  for  this  POI  consists  of  the  brigade 
commander,  brigade  XO,  exercise  director,  exercise  administrator,  workstation  operators, 
interactors,  and  observers 

The  WST  portion  of  the  T^  POI  was  developed  to  meet  requirements  of  field  commanders  as 
expressed  in  the  needs  assessment.  They  asked  for  an  easily  exportable  system,  which  would  not 
require  a  lot  of  training  to  learn  to  use.  The  developers  assumed  that,  since  the  Army  standard 
desktop  is  an  Intel  processor  running  Microsoft  Windows  (95®  or  NT®),  any  software  developed 
to  nm  in  this  environment  would  require  minimal  outside  assistance  to  use.  Microsoft  Help®  was 
chosen  to  develop  the  tutorials  for  the  following  reasons:  (a)  the  appearance  of  information 
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matches  that  with  which  users  of  the  Windows*  environment  are  familiar,  (b)  it  is  simple  to 
modify,  and  (c)  tutorials  developed  with  this  are  easy  to  distribute  and  use. 

Currently,  the  WST  package  consists  of  two  separate  programs.  The  WST  1  program  is  for 
use  at  homestation,  while  the  WST  2  program  is  for  training  on  site.  The  WST  1  program 
provides  ample  system  background  to  permit  the  workstation  operator  to  arrive  on  site  and  work 
through  the  full  tutorial.  (Detailed  UNDC  implementation  instructions  are  not  contained  in  the 
WST  1  program,  since  practicing  them  requires  the  Sun  systems  to  be  present.)  On  the  first  day 
the  unit  utilizes  the  SGT  system,  the  WST  2  tutorial  allows  workstation  operators  to  practice 
skills  at  their  respective  workstations. 

The  tactical  materials  consist  of  the  division  and  brigade  OPORDs  with  overlays,  and  a 
brigade  SOP  extract.  This  is  the  only  area  where  a  unit  may  make  appropriate  adjustments  to  fit 
their  particular  unit  specifications. 

The  exercise  training  materials  were  developed  to  support  the  pre-execution  staff  huddle. 
This  is  a  “last  minute”  meeting  where  the  XO  gathers  the  main  CP  staff  to  ensure  that  each  staff 
section  understands  the  commander’s  vision,  expectations,  and  intent  for  the  battle  and  for  each 
individual  BOS.  It  normally  lasts  only  a  few  minutes  and  involves  the  XO  and/or  BC,  and  at  a 
minimum  the  staff  section  officers-in-charge/non-commissioned  officers-in-charge.  In  the  SGT 
environment  it  occurs  near  the  main  CP  SitMap.  The  XO  and  staff  are  provided  the  maps  and 
overlays  needed  after  the  individual  staff  section  preparation  activities  are  completed  but  prior  to 
the  exercise  commencing.  Developers  organized  a  T^  pre-execution  staff  huddle  guide.  This 
guide  is  a  series  of  BOS-specific  questions  based  on  information  found  in  the  DST  and  the 
synchronization  matrix  of  the  OPORD.  It  provides  the  XO  or  BC  with  information  to  personally 
verify  that  his  main  CP  staff  is  well  grounded  in  the  operation  and  is  alert  for  specific  cues 
throughout  the  battle. 

Review  Janus  Video  Tapes  [71 

As  an  important  step  in  the  exercise  definition  process,  the  Team  executed  a  Janus  version  of 
the  area  defense  exercise,  as  dictated  by  current  tactics  and  doctrine.  The  exercise  was  saved  and 
played  back  for  review.  A  video  tape  of  the  exercise  was  also  made  and  reviewed  in  detail.  This 
process  provided  valuable  insight  into  the  timing  of  events  and  activities  within  the  mission.  It 
also  allowed  the  Team  to  verify  the  time/distance  factors  of  the  forces  on  the  battlefield  and 
provided  additional  input  in  determining  tactical  outcomes  and  events.  This  exercise  served  as  a 
basis  for  determining  battle  damage  assessment  (BDA)  and  other  related  battlefield  information 
in  SGT. 

Identifi^  Critical  Events  fSaj 

Each  exercise  was  custom  constructed  to  set  the  stage  for  the  staff  to  perform  the  desired 
learning  objectives.  The  Team  used  worksheets  to  identify  and  record  the  exercise’s  critical 
events.  Exercise  critical  events  are  defined  as  actions  in  the  tactical  scenario  that  cue  battle  staff 
functions  in  support  of  the  command  and  control  (C^)  cycle  (Department  of  the  Army,  1995b). 
The  Team  classified  an  exercise  critical  event  as  an  action  which  causes  employment  of  DST 
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criteria,  requires  a  staff  BOS  activity  on  the  staff  synchronization  matrix,  or  answers  a  CCIR. 
First,  the  exercise  author  listed  each  staff  action  within  the  exercise,  which  staff  sections  were 
affected,  and  whether  or  not  the  action  supported  a  DST  criterion.  Second,  the  author  selected 
staff  synchronization  activities  that  related  to  those  staff  sections  that  comprised  part  of  the 
training  audience,  but  not  having  DST  supporting  actions.  Next,  the  author  linked  each  event  to 
an  exercise  learning  objective.  Last,  the  author  combined  or  deleted  critical  events  as  necessary, 
stating  the  rationale  for  doing  so.  The  resulting  list  became  the  exercise  critical  event  list,  which 
drove  the  development  of  the  message  traffic. 

Identify  Key  Messages  f8b1 

The  Team  modified  the  area  defense  message  traffic  database  from  the  prior  SGT  to  support 
the  design  concepts  of  the  SGT  exercises.  This  entailed  making  additions,  deletions,  and  content 
changes.  The  critical  events  identified  within  each  exercise  had  to  be  supported  tactically.  The 
exercises  had  to  be  built  around  the  desired  learning  objectives  identified  for  each  exercise, 
ensuring  that  each  staff  section  received  the  quantity  and  quality  of  messages  that  they  could 
realistically  expect  to  receive  from  appropriate  BOS-specific  sources  within  the  prescribed 
tactical  scenario.  Once  the  exercise  messages  were  finished,  developers  systematically 
determined  each  exercise’s  key  messages;  these  provided  the  cues  which  defined  the  exercise’s 
critical  events  in  a  systematic  manner.  This  required  the  developers  to  run  several  quality 
assurance  dry  runs  to  ensure  all  participants  were  exposed  to  a  flowing  exercise.  These  key 
messages  were  also  used  to  develop  the  observer  checklists,  to  tightly  focus  the  observer  on  what 
staff  actions  should  be  happening  at  a  given  point  in  the  exercise. 

Develop  Staff  Support  Process  Models  [91 

The  staff  support  process  model,  or  “mind  map”  (see  Figure  4),  is  a  pictorial  or  graphic 
displaying  an  exercise’s  critical  events,  a  brief  summary  of  the  key  messages  that  develop  each 
critical  event,  and  a  flow  diagram  showing  (in  terms  of  the  learning  objectives)  the  staff  sections’ 
expected  distribution  of  the  information  contained  in  the  key  messages.  These  “mind  maps” 
were  developed  for  each  exercise  for  two  purposes;  (a)  to  use  as  a  flow  chart  or  “story  board”  for 
the  Team  in  the  creation  of  the  exercises  and  tables  and  (b)  to  serve  as  job  aids  for  the  observers 
to  help  them  understand  the  Hierarchical  Input  Process  Output  sequence  or  cause  and  effect 
chain  of  the  exercise. 


Exercise  Materials  Development 
Develop  Initial  Message  Traffic  [8c1 

The  Team  used,  as  a  baseline  document,  the  consolidated  message  list  for  the  brigade  area 
defense  scenario  developed  during  the  previous  SGT.  In  the  developmental  stages  of  the  earlier 
effort,  that  Team  employed  a  map  exercise,  augmented  by  a  videotape  of  a  Janus  run,  and  the 
DST  and  synchronization  matrix  to  determine  the  type  of  message  traffic  that  should  be  received 
by  the  training  staff  from  subordinate,  adjacent,  and  higher  units  or  staffs.  The  Janus  videotapes 
provided  the  development  team  with  verified  activities,  times,  locations,  and  strengths  of  enemy 
and  friendly  units.  They  collected  this  information  into  a  message  summary  database.  This 
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database  included  time,  message  type,  a  key  message  identifier  code  (visible  to  developers  only), 
message  originator,  addressee,  radio  net,  and  a  message  summary.  Based  on  the  message 
summary  database,  the  Team  constructed  the  message  traffic  database. 


CUES 


PROCESS 


OUTPUT 


LEGEND 

ACA 

Airspace  CoorcS nation  Area 

CAS 

Close  Air  Support 

COA 

Course  of  Action 

MBA 

Main  Battle  Area 

NAl 

Named  Area  of  interest 

PL 

Phase  Line 

SACC 

Subordinate  and  Adjacent 

Command  &  Control 

SEAD 

Suppression  of  Enemy  Air  Defense 

TF 

Task  Force 

TR 

Tank  Regiment 

Figure  4.  Example  staff  support  process  model  (“mind  map”). 
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Use  Scattergraph  for  Message  Distribution  [Sd] 


Providing  a  constant  message  flow  to  maintain  the  appropriate  level  of  challenge  within  the 
exercise  was  an  important  concept  for  the  current  effort.  The  area  defense  mission  was  analyzed 
by  breaking  it  down  into  five  minute  increments  and  reviewing  each  portion  to  determine  the 
distribution  of  message  traffic  across  all  the  brigade  nets  and  to  determine  a  measure  of  the 
brigade  staff  activity.  Both  the  network  and  the  recipient  were  annotated  to  ensure  a  logical, 
normalized,  message  flow  and  an  even  distribution  of  messages  within  the  framework  of  the 
tactical  situation.  The  Team’s  goal  was  to  avoid  overloading  nets  and  personnel  and  to  provide 
additional  message  traffic  where  gaps  appeared,  especially  those  that  cued  critical  events.  In 
addition,  this  allowed  the  Team  to  review  and  compare  key  message  timing  against  total  message 
flow,  relative  to  staff  activity,  learning  objectives,  and  required  staff  actions. 

Develop  Strength  Charts  and  Overlays  [10] 

The  Team  initially  used  the  set  of  charts  in  Field  Circular  (FC)  71-6  (Department  of  the 
Army,  1985)  as  a  baseline  to  determine  what  information  charts  each  staff  section  would 
maintain.  The  Team  developed  a  combat  vehicle  strength  chart  that  reflected  companies  and 
separate  platoons.  Upon  developing  the  message  traffic  for  each  exercise,  exercise  authors 
prepared  two  charts  for  each  staff  section;  the  start  of  exercise  (STARTEX)  combat  strength 
situation  and  the  ENDEX  combat  strength  situation  for  those  umts  that  each  section  was 
responsible  for  tracking.  The  beginning  information  was  obtained  by  reviewing  the  Janus 
videotape  and  the  message  database  up  to  the  start  of  the  exercise.  The  ENDEX  charts  were 
prepared  by  adjusting  the  STARTEX  chart  based  on  message  traffic  during  the  exercise.  Where 
needed,  messages  were  added  to  the  exercise  message  database  to  reflect  appropriate  combat 
gains  and  losses.  The  development  team  then  produced  exercise  graphics  to  include  STARTEX 
and  ENDEX  friendly  and  enemy  unit  locations,  status  of  friendly  obstacle  plan,  and  the  status  of 
R&S  plan.  The  exercise  author  determined  the  appropriate  information  for  each  overlay  and 
created  the  overlay  for  the  workstation  map  display.  Since  brigades  track  company-level  forces 
and  higher,  the  charts  were  structured  to  reflect  that  level  of  interest. 

After  surveying  several  references  and  talking  with  a  number  of  doctrinal  subject  matter 
experts  (SMEs),  the  Team  found  that  no  standardized,  doctrinal  methodology  for  tracking 
friendly  and  enemy  combat  power  existed.  The  method  used  by  umts  tended  to  be  tailored  to 
individual  preferences.  After  examining  various  examples  of  battle  strength  tracking 
presentation  methods,  including  ATCCS  ‘pie-charts’  or  ‘18  wheelers’,  and,  following 
experimentation  with  a  number  of  different  color  and  graphic  combinations,  the  developers 
produced  a  hybrid  version  of  a  power  chart  that  featured  color-coding  to  allow  the  brigade  staff 
to  track  friendly  force  vehicles,  personnel,  fuel,  and  ammo.  The  developer  used  a  similar  color¬ 
coding  system  for  enemy  forces,  but  orgamzed  the  enemy  BDA  tracking  to  correspond  with  the 
likely  tactical  employment  (e.g.,  forward  security  element,  main  body). 


33 


Develop  Staff  Section  Preparation  Materials  [Ha] 


The  SGT  project’s  automated  staff  section  preparation  package  evolved  from  earlier  SGT 
work.  With  no  standing  0/C  team  for  the  current  effort,  the  developers  made  staff  section 
preparation  a  machine-delivered,  mandatory  process.  This  ensured  that  the  staff  sections 
completed  all  preparation  activities.  The  process  included  a  crosswalk  of  the  exercise  learning 
objectives  and  the  applicable  U.S.  Army  Training  and  Evaluation  Program  (ARTEP)  MTP  tasks, 
a  review  of  the  CCIR,  DST,  and  staff  synchronization  matrix  (with  those  critical  portions 
highlighted),  and  a  discussion  of  how  staff  success  could  be  determined,  in  terms  of  the  tactical 
situation  and  the  learning  objectives. 

Develop  Exercise  Multimedia  Preview  f  12b1 

In  the  absence  of  a  standing  0/C  team,  an  automated  version  of  the  pre-exercise  briefing  was 
created  using  Macromedia  Director™.  Based  on  the  tactical  materials,  this  multimedia  preview 
provided  a  complete  overview  of  the  general  tactical  situation,  as  well  as  the  selected  exercise, 
and  enhanced  the  immersion  of  the  staff  in  the  activities  and  actions  to  follow. 

Feedback  Material  Development 
Begin  AAR  Material  Development  [11] 

At  the  same  time  as  exercise  material  development  was  taking  place,  work  on  the  structure 
and  content  of  the  AAR  feedback  materials  was  begun.  Based  on  staff  learning  objectives,  the 
creation  of  these  materials  was  primarily  concerned  with  reformatting  and  automation,  applying 
the  process  previously  designed  for  use  by  the  0/C  training  team. 

Develop  Section  and  Exercise  After  Action  Review  Materials  fl  la,  lib] 

With  the  exception  of  the  staff  transition  table  (which  requires  only  the  staff  section  AAR), 
all  exercises  require  each  staff  section  to  participate  in  an  individual  staff  section  AAR  and  an 
exercise  AAR,  involving  the  entire  training  audience.  The  AAR  presentation  formats  for  the 
section  and  exercise  AARs  were  developed  to  facilitate  a  self-administered  “discovery”  session 
using  automated  data.  The  automated  AAR  process  requires  the  staff,  both  as  individual  sections 
and  as  a  consolidated  group,  to  conduct  a  thorough  self-assessment  by  closely  focusing  on  their 
use  of  the  OPORDs  products  (specifically,  the  CCIR,  DST,  and  staff  synchronization  matrix)  and 
execution  of  the  learning  objectives. 

Develop  Required  Print  Material  [ba] 

Some  exercise-specific  training  tools  for  the  trainers  and  training  staff  are  maintained 
electronically  and  printed  on-site  by  the  exercise  administrator  on  the  day  of  training  execution. 
This  saves  the  unit  preparation  time  in  gathering  these  materials  from  various  sources,  ensures  all 
proper  materials  are  available  to  appropriate  trainers,  and  reduces  the  bulk  of  the  Brigade  Tables 
book.  The  development  decision  regarding  which  tools  should  be  printed  on-site  was  based  on 
the  requirement  to  provide  only  the  tools  that  would  most  likely  be  available  in  a  “real  world” 
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situation.  The  Team  determined  which  tools  applied  to  each  staff  section  or  training  support 
personnel.  When  the  paper-based  materials  are  printed  on-site,  they  are  automatically  collated 
into  packets  appropriate  for  each  user,  as  indicated  in  Table  8. 

Table  8 

On-Site  Printed  Tools 


Training  Material 

Training 

Team 

User 

Staff 

in  Training 

Training  team  roster 

y 

AAR  guidelines/outline 

✓ 

Exercise  Director  checklist 

✓ 

Interactor  RFI  guidelines 

✓ 

Exercise  message  list 

y 

Staff  Support  Process  Model 

y 

STARTEX  combat  strength  charts 

y 

y 

Action  Plan  worksheet  (XO) 

y 

y 

Signal  Operating  Instructions  Extract 

y 

y 

Pre-Execution  Staff  Huddle  Guidelines  (XO) 

✓ 

✓ 

X 

Staff  Synchronization  Matrix 

Decision  Support  Matrix 

y 

y 

y 

Support  Requirements 

To  complete  the  TSP  development  it  was  necessary  to  specify  the  support  requirements  for 
the  new  SGT  exercises.  Unit  participants  for  the  current  effort  include  a  staff  to  be  trained  and  a 
training  team.  The  training  team  is  composed  of  five  or  six  unit  personnel  selected  by  the 
brigade  commander  to  perform  administrative,  observer,  and  mentoring  functions  (see  Table  5). 
If  a  brigade  commander  has  additional  personnel  he  wants  to  involve  as  members  of  the  training 
team,  they  could  perform  mentoring  functions  according  to  their  specific  expertise  (e.g.,  FS,  EN). 

To  support  brigade  staff  training  for  the  main  CP,  a  minimum  of  seven  SGT  workstations 
are  required  (see  Figure  2).  As  mentioned  previously,  workstations  are  linked  by  a  LAN  so  they 
can  send  and  receive  information  to  and  from  one  another.  Workstations  are  used  as  follows: 
four  workstations  are  allocated  for  staff  sections  for  each  exercise,  one  workstation  supports  the 
exercise  administrator  with  network  status,  one  workstation  controls  the  exercise  and  responds  to 
staff-generated  message  traffic  sent  to  higher  HQ,  and  one  workstation  responds  to  staff- 
generated  message  traffic  sent  to  subordinate  and  adjacent  units.  Personal  computers  are  co¬ 
located  with  the  Sun  workstations  for  the  S3,  S2,  FSE,  and  EN  sections.  They  are  used  to 
conduct  staff  section  preparation  before  execution,  refer  to  tactical  training  aids  during 
execution,  and  facilitate  staff  section  AARs  after  execution.  The  SGT  site  also  has  a  large-screen 
display  that  is  networked  to  the  workstations  via  the  LAN.  It  is  used  prior  to  execution  to  show 
exercise  previews,  during  execution  as  the  SitMap,  and  after  execution  for  the  exercise  AARs. 
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There  are  two  printers  located  in  the  SGT  area.  One  is  in  the  CP  area  to  emulate  a  imit  fax 
machine.  It  handles  all  messages  as  outlined  in  tactical  SOP  going  to  mobile  subscriber 
equipment.  The  other  printer  is  near  the  interactor  workstations.  Its  functions  are  to  print 
exercise  training  materials  prior  to  execution  and  to  emulate  higher  and  subordinate  HQs’  fax 
machines  during  execution. 

The  latter  emulates  the  fax  capabilities  in  a  tactical  CP.  Reports  that  would  come  into  a 
tactical  CP  are  the  same  reports  that  will  come  off  the  fax  during  staff  group  training.  During 
execution,  appropriate  staff  will  get  a  fax  alert  message  on  their  workstation. 
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FORMATIVE  EVALUATION:  METHODS  AND  RESULTS 


The  purpose  of  formative  evaluation  was  to  monitor  and  track  program  execution,  consistent 
with  the  program  objectives.  Specifically  the  SGT  formative  evaluation  focused  on  the 
integration  of  theoretical  constructs  with  off-the-shelf  technology  as  applied  to  meet  the  staff 
training  needs  established  in  the  needs  assessment.  It  involved  the  following  levels:  internal 
review  and  testing,  prototype  review,  a  pilot  test,  and  an  implementation  trial. 

The  FE  team  used  the  following  methods  to  collect  data:  observer  checklists,  participant 
questionnaires  (see  Appendix  C),  and  “hot  washes”  or  group  interviews  for  data  collection.  As 
the  tactical  tables  were  executed,  designated  members  of  the  Team  acted  as  observers  and 
collected  data  on  occurrences  related  to  each  of  the  component  parts  of  the  SGT  exercise: 
exercise  preview,  staff  section  preparation,  execution,  section  AAR,  and  exercise  AAR.  These 
data  were  augmented  with  participant  questionnaires  administered  after  each  table  (table  specific) 
and  a  final  questionnaire  (which  asked  the  individuals  to  rate  the  overall  project).  For  the  “hot 
washes”  the  participants  were  divided  into  three  groups:  (a)  enlisted,  (b)  officer  (less  the 
command  group),  and  (c)  the  command  group  (i.e.,  the  S3,  XO,  and  commander). 

This  chapter  focuses  on  the  results  and  suggestions  from  each  of  the  three  externally 
supported  formative  evaluation  events — ^the  prototype  review,  the  pilot  test,  and  the  trial — ^which 
required  non-SGT  team  members  to  review  products  or  take  part  in  the  SGT  training.  The 
internal  formative  evaluation  activities  were  discussed  earlier  in  this  report  in  the  TSP 
Development  Methodology  Chapter. 


Prototype  Review 
Description  of  the  Event 

The  prototype  review  was  conducted  as  a  user  panel;  it  required  24  hours  over  four  days. 
The  purpose  of  this  review  was  for  the  participants  to: 

1 .  Identify  problems  related  to  the  TSP. 

2.  Provide  suggestions  for  revisions. 

3.  Comment  on  the  usability  and  acceptability  of  the  training  program  content. 

4.  Provide  opinions  on  various  concepts  and  approaches  to  the  training  not  yet  implemented 
in  the  prototype  materials. 

In  most  prototype  reviews,  the  panel  examines  all  phases  and  materials  for  a  single  exercise. 
However,  in  this  project,  which  relied  on  the  heavy  use  of  technology  to  implement  the  training 
strategy,  a  linear  approach  was  not  cost  effective.  Instead,  the  panel  reviewed  the  design 
concepts  and  prototypes  of  the  materials  that  were  to  be  developed  for  use  on  the  computers  and 
PDAs.  Specifics  are  discussed  in  the  paragraphs  that  follow.  The  purpose  of  this  early  look  was 
to  assure  the  Team  that  the  development  direction  was  the  correct  one  and  would  meet  the  needs 
of  the  training  audience. 
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The  formative  evaluation  plan  called  for  a  prototype  review  by  four  military  personnel.  The 
requirements,  as  well  as  the  ranks  and  qualifications  of  the  review  participants,  are  shown  in 
Table  9.  While  only  one  active  duty  individual  was  provided,  the  experience  level  and  expertise 
of  all  participants  exceeded  what  was  requested.  The  participants  provided  valuable, 
independent  insights,  since  all  had  worked  on  other  training  projects  and  none  were  directly 
involved  v^th  the  SGT  project.  Their  input  was  gathered  through  a  series  of  informal  meetings 
and  briefings,  with  an  analysis  based  on  application  of  the  small  group  instruction  methodology. 

Table  9 

Panel  Members 


Requirement 

Participant 

Lieutenant  Colonel  -  Maneuver  (Armor  or 
Infantry) 

Lieutenant  Colonel  -  Infantry 
(Training  Brigade  XO) 

Major  -  Military  Intelligence 
(with  a  tactical  intelligence  specialty) 

Lieutenant  Colonel  (Retired)  -  (Military 
Intelligence) 

Major  -  Field  Artillery 

Brigadier  General  (Army  National  Guard)  - 
Former  Field  Artillery  Brigade  Commander 

Colonel  (Retired)  -  Field  Artillery 

Major  -  Engineer 

Lieutenant  Colonel  (Retired)  -  Engineer 
(employed  by  contractor) 

The  following  paragraphs  summarize  significant  points  made  by  the  panel  members  during 
the  prototype  review.  Those  components  of  the  SGT  training  system  which  are  implemented  on 
the  computer — and  therefore  reviewed  conceptually — ^have  this  fact  noted  in  their  title.  A 
summary  of  the  changes  made  to  the  TSP  materials  based  on  this  process  is  found  at  the  end  of 
this  subsection. 


Pre-Exercise  Material 


Training  Guide 

The  user  panel  expressed  concern  about  the  purpose  and  method  of  delivery  of  the  Training 
Guide.  The  panel  suggested  that  the  information  necessary  for  the  commander  and  key  unit 
members  be  put  in  electronic  form  for  computer  delivery. 

Train-the-Trainer  Program 

The  panel  members  saw  a  pressing  need  for  a  T^  program.  They  expressed  concern  about 
how  a  unit  would  be  able  to  conduct  the  program  on  their  own.  They  saw  the  development  of  a 
successful  T^  program  as  a  critical  challenge  for  the  developers. 
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Exercise  Preparation  Material 


Exercise  Preview  (Computer  Delivery  During  Training) 

Panel  members  saw  a  prototype  multimedia  exercise  preview'  that  was  created  during  the 
previous  SGT  effort  and  were  given  the  script  for  the  exercise  they  would  be  reviewing.  The 
revised  exercise  preview  was  to  have  more  enemy  information,  a  review  of  the  commander’s 
CCIR,  and  a  discussion  of  the  next  phase  of  the  plan.  The  panel  agreed  that  these  changes  were 
needed. 

Section  Preparation  (Computer  Delivery  During  Training) 

The  panel  was  provided  with  computer-based  S2  staff  section  preparation  material  to  review. 
The  in-depth  content  review  of  this  material  was  conducted  by  the  military  intelligence  (MI) 
member  of  the  panel.  He  provided  additional  information  on  intelligence  asset  feeds  to  be  added 
and  recommended  that  more  information  be  included  on  the  map  concerning  the  enemy  situation. 

Pre-Exercise  Staff  Huddle 


At  this  point  in  the  development  cycle  the  pre-exercise  staff  huddle  was  called  a  mini- 
rehearsal.  The  panel  member  discussion  centered  on  the  training  staff  s  need  to  thoroughly 
understand  the  OPORD.  Panel  members  stated  that  a  staff  rehearsal  or  OPORDs  brief  should  be 
a  requirement  prior  to  conducting  an  exercise.  They  also  suggested  that  structured  guidance  be 
provided  to  the  unit  on  the  intent  of  the  pre-exercise  staff  huddle  and  how  it  should  be  conducted. 
These  suggestions  were  incorporated  into  the  TSP  materials. 

Exercise  Execution  Material 


Message  List  (Computer  Delivery  During  Training) 

The  FS  and  MI  panel  members  made  several  suggestions  to  improve  the  message  lists  that 
were  provided  to  them.  This  included  more  information  on  positioning  and  tracking  of  FA 
assets,  and  the  inclusion  of  additional  MI  units  to  provide  standard  intelligence  reporting. 

Observer  Checklist  (Computer  Delivery  During  Training) 

The  draft  staff  observer  checklist  was  reviewed.  The  Team  explained  to  the  panel  that  the 
checklist  would  be  placed  on  a  PDA  that  would  alert  the  observer  when  and  what  to  look  for,  as 
well  as  provide  a  place  to  record  observed  behavior.  The  user  panel  voiced  concern  about  the 
need  for  a  back-up  system  if  the  PDA  was  non-operative.  They  stated  that  virtually  anyone 
would  be  capable  of  being  a  staff  observer  using  the  PDA  with  the  observer  checklist  and  cues 
described.  Without  the  PDA,  the  observer  would  have  to  be  an  experienced  staff  officer  who 
was  familiar  with  the  exercise. 
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Coaching 


The  user  panel  felt  that  specific  guidance  on  when  and  how  much  to  coach  should  be 
provided  to  the  observers.  At  this  point  in  the  development,  specific  coaching  criteria  had  not 
been  established.  Based  on  this  guidance,  specific  coaching  questions  were  developed  for  each 
key  message  (e.g.,  reminders  on  the  significance  of  reports;  hints  to  cue  specific  activities).  This 
encouraged  the  participants  to  correctly  perform  their  staff  actions. 

Interactor  Cell 

The  Team  presented  only  the  concept  for  the  interactor  cell.  The  panel  members  were 
concerned  about  whether  a  brigade  would  be  able  to  find  individuals  capable  of  meeting  the 
requirements  of  the  interactor  cell.  They  felt  that  the  individuals  in  this  cell  would  have  to  be 
highly  qualified  and  thoroughly  familiar  with  the  exercise. 

After  Execution  Feedback  Material 


For  the  after  execution  feedback  material,  the  Team  presented  only  the  concepts  since  all 
these  components  were  to  be  computer  delivered  and  were  still  in  development.  For  many  of 
these  ideas,  the  Team  used  the  panel  as  a  sounding  board  to  assist  in  determining  the  actual 
structure  of  the  feedback. 

Section  AAR  (Computer  Delivery  During  Training) 

The  user  panel  understood  the  concepts  behind  the  “WOO”  and  the  “staff  fingerprint”  but 
provided  few  comments  or  suggestions  for  these  two  tools.  One  concept  presented  by  the  Team 
was  the  idea  of  having  each  section  rate  the  performance  of  the  other  sections  in  terms  of 
learning  objective  performance  on  certain  critical  messages.  The  idea  was  for  the  sections  to 
exchange  this  information  in  an  attempt  to  look  at  how  one  section’s  performance  impacted  on 
the  performance  of  other  sections.  The  user  panel  voiced  serious  concerns  about  this,  believing  it 
would  be  interpreted  as  a  “peer  rating.” 

Exercise  AAR  (Computer  Delivery  During  Training) 

The  user  panel  voiced  concern  about  the  amount  of  information  the  training  team  would 
have  to  process  to  prepare  the  exercise  AAR  and  the  time  allocated  to  have  the  information 
processed  and  incorporated  into  the  AAR. 

Commander’s  Staff  Training  Profile  (Computer  Delivery  During  Training) 

The  user  panel  reinforced  the  views  of  the  commanders  interviewed  in  the  needs  assessment 
that  the  feedback  had  to  be  short,  focused  and  timely.  The  user  panel  also  expressed  a  concern 
that  information  contained  in  the  profile  might  be  used  for  unit  evaluation  reports. 
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TSP  Modifications 


The  user  panel’s  concerns,  comments,  and  suggestions  led  to  the  following  SGT  team 
actions: 

1 .  Revising  the  Training  Guide.  It  was  decided  that  the  Training  Guide  would  not  be  put 
in  electronic  form.  If  the  SGT  were  to  be  fielded,  the  electronic  format  should  be  used; 
however,  the  fiinds  available  during  this  project  were  only  adequate  to  electronically 
implement  those  components  which  directly  contributed  to  the  proof  of  principle  of  the 
small  group  training  methodology. 

2.  Expanding  the  number  and  content  of  T^  packages  to  include  detailed  directions  for  the 
commander,  XO,  exercise  director,  exercise  administrator,  workstation  operators, 
interactors  and  observers.  Additional  job  aids  were  developed  to  be  printed  before  each 
individual  exercise. 

3.  Developing  new  multimedia  exercise  previews  more  tightly  linked  to  the 
learning  objectives. 

4.  Establishing  detailed  coaching  criteria  and  incorporating  these  into  the  PDA 
message  sets  to  assist  inexperienced  observers. 

5.  Developing  detailed  observer  checklists  and  coaching  questions  and  installing  them  on 
the  PDAs.  Additionally,  printed  checklists  were  available  from  the  electronic  format 
(Microsoft  Excel®). 

6.  Examining  the  staff  section  preparation  material  closely.  The  comments  fi'om  the  S-2 
reviewer  were  incorporated  into  the  material,  and  the  time  allocated  in  the  training 
schedule  to  conduct  the  staff  section  preparation  was  expanded  to  30  minutes. 

7.  Refining  the  exercise  message  based  on  the  comments  of  the  FS  and  the  S2  review 
panel  members. 

8.  Incorporating  structured  guidance  on  how  to  conduct  a  pre-exercise  staff  huddle  in  the 
T^  and  the  pre-exercise  print  materials  for  the  XO  and  commander. 

9.  Developing  additional  job  aids  for  the  interactors  (e.g.,  ground  truth  overlays  for 
every  five  minutes  of  battle). 

10.  Dropping  any  attempt  to  implement  the  rating  of  other  sections  during  the  staff 
section  AAR. 

11.  Adding  additional  structure  to  the  exercise  AAR  to  streamline  its  implementation. 

Pilot 

An  Army  National  Guard  unit  was  the  test  unit  and  the  training  schedule  was  restructured  to 
fit  into  a  Multiple  Unit  Training  Assembly  4  (MUTA  4)  weekend.  This  meant  that  the 
evaluation  of  unit  preparation  materials  had  to  be  conducted  at  the  unit’s  homestation  prior  to  the 
pilot. 


Formative  Evaluation  Instruments 


The  current  SGT  formative  evaluation  used  techniques  similar  to  previous  SGT  efforts  (see 
Roger  et  al.,  1998).  However,  the  SGT  participant  questions  for  the  current  effort  were  all  based 
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on  a  Likert-type  scale.  The  scale  ranged  from  -3  to  -i-3.  For  questions  dealing  with  effectiveness 
or  appropriateness  of  the  training,  the  scales  were  anchored  by  the  terms  extremely  ineffective, 
neutral,  and  extremely  effective.  Far  too  easy,  about  right,  or  far  too  difficult  were  used  when  the 
questions  dealt  with  the  level  of  exercise  difficulty.  In  all  other  cases  the  anchors  used  were 
definitely  no,  neutral,  or  definitely  yes. 

Evaluation  of  Unit  Preparation  Materials 

This  evaluation  was  accomplished  at  the  unit’s  armory  during  a  weekend  drill.  This  was 
done  approximately  three  weeks  before  the  unit  arrived  for  the  pilot. 

Tactical  Orders 


The  tactical  orders  consisted  of  the  OPORD  for  a  brigade  defense  operation.  The  doctrinal 
accuracy  of  the  OPORD  had  already  been  evaluated  and  approved  during  prior  SGT  efforts. 

Workstation  Operator  Training/Familiarization 

The  unit  had  provided  four  individuals  to  go  through  the  WST/familiarization  at  their  home 
station.  These  individuals  received  the  following  instructions; 

Run  through  the  contents  until  you  are  familiar  with  them.  Note  that  underlined  green 
words  are  hyperlinked,  and  that  the  pictures  have  text  attached  to  them  that  will  appear 
when  the  cursor  turns  into  a  little  hand  and  you  click  on  it. 

With  software  mounted  on  unit  computers  and  this  limited  oral  instruction,  three  individuals 
had  no  difficulty  going  through  the  screens  and  worked  independently  through  the  material  as  it 
was  designed.  One  individual,  however,  had  never  used  a  computer  before,  and  had  difficulty 
using  the  mouse. 


Purpose  of  the  Pilot 

The  purpose  of  the  pilot  was  to  evaluate  the  exercises  and  materials  scheduled  for  use  in  the 
trial.  There  was  only  one  exception — ^the  commander’s  staff  training  profile.  It  was  not  used  in 
the  pilot  because  the  software  for  this  component  was  not  yet  ready.  The  pilot  followed  the 
schedule  shown  in  Appendix  E. 


Pilot  Results 


The  data  collection  effort  covered  many  detailed  aspects  of  the  training  project.  At  this 
point  the  major  concern  was  to  determine  if  the  exercises  and  materials  were  effectively  teaching 
the  correct  tasks.  In  addition,  the  Team  wanted  to  ensure  that  the  exercises  were  at  a  level  of 
difficulty  which  was  challenging  enough  to  retain  the  interest  of  the  training  audience  while  not 
causing  them  to  lose  the  self-confidence  needed  for  success. 
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On  Site  Workstation  Operator  Training 


While  the  officers  reviewed  the  tactical  OPORDs,  the  workstation  operators  completed  30 
minutes  of  computer-based  instruction  (CBI).  Upon  completion  of  this  training,  the  formative 
evaluation  team  administered  questionnaires  to  the  participants.  The  questions  dealt  with  the 
effectiveness  and  appropriateness  of  the  WST.  All  four  operators  rated  the  training  positively. 
The  two  highest  ratings  were  from  the  two  operators  who  rated  themselves  as  very  proficient  and 
knowledgeable  on  computers.  All  respondents  felt  they  needed  additional  practice.  Three  of  the 
four  felt  they  needed  1  V2  to  2  hours  of  additional  practice.  The  other  individual  felt  he  needed  an 
additional  V2  to  1  hour.  This  was  the  person  who  rated  himself  as  the  most  computer  proficient 
member  of  the  group. 

Level  of  Exercise  Difficulty 


The  “crawl,  walk,  run”  concept  dictated  that  the  exercise  represent  increasingly  more 
complex  learning,  with  no  stage  of  training  either  so  difficult  that  individuals  felt  overly 
challenged  or  so  easy  that  they  became  bored.  As  a  result,  the  exercises  were  restructured.  To 
measure  this  dimension,  the  participant  questionnaires  administered  after  each  of  the  exercises 
contained  questions  on  the  difficulty  of  the  exercises  conducted  for  both  the  individual  and  the 
staff  section.  The  results  are  shown  in  Figure  5. 
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**N=14  (EN  Section  not  included) 


Figure  5.  Mean  participant  ratings  of  exercise  difficulty  (scale  range  from  -3  to  +3)  in  the  pilot. 

It  was  the  belief  of  the  SGT  team  that  the  difficulty  rating  of  the  exercises  should  be  in  the 
positive — close  to  one  (1) — ^to  meet  the  design  criteria  and  maintain  the  challenge  level  of  the 
exercise.  With  the  pilot  participants,  the  means  approached  the  intended  difficulty  levels.  As 
was  noted  earlier  in  this  section,  the  workstation  operators,  when  surveyed  before  any  training 
was  conducted,  felt  that  they  required  additional  training  to  be  proficient.  The  data  collected 
from  these  individuals  indicated  that  their  concerns  may  have  been  premature;  across  the  board 
they  rated  the  exercises  as  less  difficult  than  the  group  as  a  whole. 
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Focus  of  Staff  Tasks  and  Learning  Objectives 


Qualitative  and  quantitative  data  indicated  that  the  exercise  kept  the  training  audience 
focused  on  the  SGT  learning  objectives  rather  than  tactical  outcomes,  and  met  the  staff  training 
needs  and  deficiencies  of  the  participant  unit. 


The  individuals  were  asked  to  indicate  on  the  participant  questioimaires  whether  they 
thought  that  the  correct  staff  tasks  had  been  trained  during  each  of  the  prototype  exercises.  The 
results  of  that  question  for  each  of  the  tables  are  shown  in  Figure  6.  These  results  indicate  that 
the  participants  felt  strongly  that  the  exercises  were  teaching  them  what  they  needed  to  know. 
Some  of  the  comments  made  provided  increased  validity  to  these  findings,  such  as  “Geared 
toward  what  our  staff  needed,”  and  “Oriented  on  appropriate  tasks  and  (it)  exercised  the  staff 
appropriately.” 


Staff  Transition  Table  Staff  Integration  Table  Command  Post  Table 
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Figure  6.  Frequency  histograms  for  participant  responses  to  the  pilot  question  “Was  the  exercise 
focused  on  the  correct  tasks  for  your  staff  section?” 

The  participants  were  questioned  directly  on  whether  they  felt  that  the  objectives  of  the 
exercise  had  been  met.  The  Team  felt  this  was  one  of  the  most  important  factors  to  assess  the 
validity  of  the  small  group  methodology  design  concepts.  The  level  of  training  of  the  various 
learning  objectives  was  assessed  using  the  participant  questionnaires  and  the  comments  provided 
during  the  hot  washes.  The  quantitative  responses  are  shown  in  Figure  7.  One  of  the  comments 
fi'om  the  hot  washes  was  provided  by  a  workstation  operator  who  said  “  I  don’t  know  if  it  will 
improve  the  ability  to  perform  the  learning  objectives,  but  it  certainly  gives  us  the  knowledge  we 
need  to  do  it.” 
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Staff  Transition  Table  Staff  Integration  Table  Command  Post  Table 


Definitely  Neutral  Definitely  Definitely  Neutral  Definitely  Definitely  Neutral  Definitely 
No  Yes  No  Yes  No  Yes 


Figure  7.  Frequency  histograms  for  participant  responses  to  the  pilot  question  “Did  the  exercise 
achieve  its  objectives?” 

Quality  of  Training  Feedback 

The  results  of  the  pilot  test  also  indicated  that  the  design  of  the  structured  AARs  at  both  the 
section  and  the  exercise  level  provided  feedback  that  the  staff  felt  was  useful.  It  can  be  seen  in 
Figure  8  that,  overall,  the  ratings  of  the  section  AARs  improved  as  the  training  progressed.  The 
differences  in  the  evaluation  of  the  section  AAR  for  the  staff  transition  table  and  section  AAR  for 
the  staff  integration  table  may  be  due  in  some  measure  to  command  emphasis.  The  staff 
transition  table  hot  wash  results  indicated  that  the  personnel  had  given  it  less  attention  because 
the  exercise  was  over.  The  individuals  had  taken  approximately  10  minutes  to  conduct  the 
section  AAR  and  had  not  correctly  filled  in  the  self  evaluation.  The  commander  and  XO  were 
notified.  When  the  unit  returned  to  execute  the  staff  integration  table,  they  were  informed  by  the 
XO  that  they  were  expected  to  properly  complete  the  section  AAR  information  and  that  he  would 
be  receiving  this  information.  The  later  evaluations  reflected  this  emphasis.  Even  comments 
reflected  the  change  in  attitude.  After  the  CP  exercise,  one  individual  wrote  “the  staff  (sic) 

AARs  draw  attention  to  mission  critical  areas.” 
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Staff  Transitiai  Table  Staff  Integratiai  Table  Ccrariand  Post  Table 
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Figure  8.  Frequency  histograms  for  participant  responses  to  the  pilot  question  “How  effective 
was  the  section  AAR?” 

The  ratings  for  the  exercise  AAR  (Figure  9)  represent  the  training  audience  perceptions  of 
the  AAR  effectiveness  in  two  instances.  The  staff  integration  AAR  was  conducted  by  the  SGT 
team,  while  the  unit  XO  conducted  the  CP  exercise  AAR.  In  fact,  the  one  -2  score  for  the  CP 
exercise  represents  the  XO’s  rating.  He  indicated  in  comments  to  the  Team  that  the  multimedia 
presentation  was  not  streamlined  enough  to  permit  easy  use  by  the  briefer.  The  other  participants 
did  not  seem  to  be  aware  of  any  problems.  One  participant  stated  that  the  CP  exercise  AAR  was 
thorough  and  objective. 
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*  The  Staff  Transition  Table  does  not  include  an  Exercise  AAR 


Figure  9.  Frequency  histograms  for  participant  responses  to  the  pilot  question  “How  effective 
was  the  exercise  AAR?” 
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Other  Insights 


The  hot  washes  and  the  questionnaires  provided  other  insights.  The  first  finding  was  that  the 
training  exercises  illuminated  certain  SOP  deficiencies.  It  was  also  found  that  the  interactor 
(higher  and  adjacent/subordinate  HQ  players)  materials  and  procedures  needed  to  be  improved 
and  that  the  number  of  messages  needed  to  be  increased  in  certain  exercises  and  certain  areas. 
Finally,  observer  checklists  were  difficult  to  use  in  tracking  the  learning  goals  of  the  exercises. 

Trial 

TSP  Modifications 

The  purpose  of  the  pilot  was  to  identify  changes  needed  prior  to  implementation  of  the  trial. 
Thus,  the  following  modifications  were  made  in  the  TSP  based  on  the  results  of  the  pilot: 

1 .  The  exercise  AAR  was  redesigned  and  streamlined. 

2.  Messages  for  the  FS,  EN  and  S2  BOS  areas  were  made  more  robust. 

3.  Interactor  (higher  and  adjacent/subordinate)  materials  were  improved  by  simplifying 
message  traffic  distribution  and  improving  workstation  overlay  capabilities. 

4.  The  software  for  the  commander’s  Staff  Training  Profile  was  implemented. 

5.  Observer  checklists  were  improved  to  allow  easier  tracking  of  learning  objectives. 

Homestation  Preparation 

The  Team  sent  two  members  to  the  unit  to  present  the  same  homestation  training  that  had 
been  provided  to  the  pilot  audience.  However,  since  there  was  no  drill  weekend  between  the 
time  the  unit  was  identified  and  the  execution  of  the  trial,  only  a  limited  number  of  personnel 
were  available.  The  Team  members  presented  a  limited  group  with  the  tactical  OPORDs  and 
installed  the  WST  package  on  computers  at  the  armory. 

Conduct  of  the  Trial 


The  training  schedule  was  tailored  to  fit  a  MUTA  5  weekend.  The  trial  was  conducted  very 
much  like  the  pilot,  with  members  of  the  SGT  team  acting  in  most  of  the  principal  trainer 
positions.  The  schedule  for  the  trial  is  shown  in  Appendix  E. 

The  imit  arrived  on  a  Friday  evening  with  the  brigade  commander,  XO  and  twelve  staff 
members.  That  evening  the  unit  participated  in  the  tactical  OPORDs  review  and  the  workstation 
familiarization  similar  to  what  the  pilot  unit  had  received  at  homestation. 

Results 


On  Site  Workstation  Operator  Training 

The  workstation  operators  completed  a  “homestation”  familiarization  on  Friday  evening.  As 
with  the  pilot  unit  workstation  operators,  this  group  rated  the  training  positively.  In  this  group, 
the  most  proficient  operator  said  he  required  no  additional  practice,  while  two  others  felt  they 
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needed  an  additional  V^  hour  of  practice.  The  final  individual  felt  he  needed  an  additional  1  to 
2  hours.  It  should  be  noted  that  the  one  individual  who  had  difficulty  was  given  additional  help. 
This  individual  consistently  felt  that  he  had  difficulty  with  the  exercises  and  rated  the  difficulty 
of  all  the  exercises  as  “far  too  hard.”  Several  individuals  commented  that  they  would  have  liked 
to  receive  instruction  30  days  before  coming  to  the  training. 

Trial 


The  questionnaires  were  administered  after  each  prototype  exercise;  however  only  two  hot 
washes  were  conducted — one  at  the  end  of  each  day.  It  was  felt  by  the  Team  that  this  would  not 
cause  significant  problems  with  data  collection  because  topics  discussed  during  hot  washes  were 
often  repetitive. 

Level  of  exercise  difficulty.  As  with  the  pilot,  participants  were  asked  to  rate  exercise 
difficulty.  Results  of  this  measure  are  in  Figure  10.  Once  again,  the  means  approached  the 
intended  difficulty  level  of  1 .  The  rated  difficulty  of  the  exercises  decreased  as  the  unit 
progressed  fi-om  table  to  table.  One  possible  explanation  for  the  decrease  in  difficulty  ratings 
was  that  the  participants  may  have  gained  proficiency  on  the  system  itself  as  training  progressed. 
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Figure  10.  Mean  participant  ratings  of  exercise  difficulty  (scale  range  firom  -3  to  +3)  in  the  trial. 

Focus  on  staff  tasks  and  learning  objectives.  The  trial  audience,  when  questioned  on  the 
correctness  of  the  staff  tasks  trained,  responded  as  shown  in  Figure  11.  These  results  again  show 
the  participants’  belief  that  the  correct  staff  tasks  were  being  trained.  One  participant  remarked 
that  the  exercise  “definitely  inereased  our  awareness  of  what  is  required,”  while  another  eited  the 
experience  as  a  “great  opportunity  to  develop  staff  interaction.” 


48 


Staff  Transition  Table 


Staff  Integration  Table  Command  Post  Table 
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Figure  11.  Frequency  histograms  for  participant  responses  to  the  trial  question  “Was  the  exercise 
focused  on  the  correct  tasks  for  your  staff  section?” 

As  with  the  pilot  participants,  the  trial  participants  were  also  questioned  on  whether  they  felt 
that  the  objectives  of  the  exercise  had  been  met.  The  responses  are  shown  in  Figure  12. 
Participants  commented  that  the  exercises  “identified  the  ‘internal  message  flow’  importance  for 
the  TOC  staff’  and  would  be  a  “valuable  tool  to  help  increase  unit  readiness.” 
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Figure  12.  Frequency  histograms  for  participant  responses  to  the  trial  question  “Did  the  exercise 
achieve  its  objectives?” 

Qnab'ty  of  training  feedback.  The  results  of  tlie  trial  indicated  that  the  design  of  the 
structured  AARs,  both  at  the  section  and  the  exercise  level,  provided  feedback  that  the  staff  felt 
was  useful. 

As  with  the  pilot,  the  section  AAR  effectiveness  ratings  improved  as  the  training  progressed 
(see  Figure  13).  These  AARs  were  often  cited  as  an  important  aspect  of  the  SGT  system. 
Participants  identified  the  AARs  as  being  “an  effective  tool  to  bring  together  the  big  picture”  and 
providing  the  opportunity  to  “allow  us  to  think  of  better  ways  to  gather  information”  in 
comments  made  on  the  questionnaires  and  during  the  hot  washes.  It  should  be  noted  when 
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considering  the  results  shown  in  Figure  13  that  more  command  group  personnel  were  available  to 
provide  data  for  the  staff  transition  table  histogram. 
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Figure  13.  Frequency  histograms  for  participant  responses  to  the  trial  question  “How  effective 
was  the  section  AAR?” 

The  histograms  in  Figure  14  represent  the  training  audience  perceptions  of  the  exercise  AAR 
effectiveness.  The  staff  integration  AAR  was  conducted  by  the  SGT  team,  while  the  unit  XO 
conducted  the  CP  exercise  AAR  with  directive  comments  made  by  the  unit  commander.  The 
only  negative  score  (-1)  in  AAR  effectiveness  came  from  the  senior  S2  analyst  during  the  staff 
integration  exercise.  He  indicated  in  comments  to  the  SGT  team  that  the  exercise  AAR  took  too 
long  to  complete,  but  he  was  otherwise  satisfied  with  the  exercise.  Other  comments  indicated 
that  the  exercise  AAR  was  very  useful. 
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The  Staff  Transition  Table  does  not  include  an  Exercise  AAR 


Figure  14.  Frequency  histograms  for  participant  responses  to  the  trial  question  “How  effective 
was  the  exercise  AAR?” 
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Other  insights.  The  hot  washes  and  the  questionnaires  provided  additional  insights  to  the 
Team.  Partieipants  felt  that  the  exercises  pushed  the  staff  to  perform  their  duties  correctly  and  in 
a  timely  fashion.  They  saw  this  as  an  opportunity  to  identify  shortcomings  in  specific  staff 
coordination  areas,  and  to  conduct  staff  integration  and  coordination  training  not  provided 
elsewhere.  Finally,  participants  remarked  positively  on  the  training  system’s  capability  to  allow 
the  staff  to  train  without  the  involvement  of  other  battalion  or  company  level  persormel. 
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LESSONS  LEARNED 


One  of  the  more  important  aspects  of  this  project  is  the  documentation  and  discussion  of 
lessons  learned  during  the  unfolding  of  the  project — ^mainly  those  issues  which  had  significant 
effects  upon  the  final  outcome  and  completion  of  products.  The  sections  of  this  chapter  are 
organized  to  follow  the  structure  of  the  project:  (a)  Analysis,  (b)  Training  Program  Design,  (c) 
Exercise  Development,  and  (d)  Project  Evaluation. 

Analysis 

Needs  Assessment 

A  key  challenge  during  the  previous  SGT  effort  was  determining  the  appropriateness  of  the 
difficulty  level  of  training.  The  issue  concerned  the  need  for  a  crawl-level  staff  training  program. 
The  needs  assessment  made  an  important  contribution  to  the  current  effort  by  confirming  the 
requirement  for  crawl-level  staff  training.  As  a  result,  the  focus  of  the  training  was  at  the  correct 
level  and  on  the  correct  objectives.  Additionally,  the  unit’s  training  and  design  constraints 
determined  if  the  needs  assessment  matched.  This  meant  that  the  unit  saw  this  training  program 
as  something  they  could  use. 

Originally,  the  current  effort  did  not  call  for  a  needs  assessment  but  the  pay-off  was 
worthwhile.  Future  efforts  like  SGT  should  start  with  an  up-to-date,  accurate  needs  assessment. 
At  a  minimum,  the  needs  assessment  must  identify  the  training  focus,  the  target  audience,  and 
training  constraints.  It  is  a  first  and  essential  step  for  establishing  a  documented  basis  for 
thorough  understanding  of  the  training  requirements.  The  results  of  the  needs  assessment 
imparted  important  aspects  to  the  design. 

Literature  Review 

As  part  of  the  front  end  analysis,  a  broad-based  literature  review  was  conducted.  Adult 
learning,  small  group  training,  theoretical  literature  related  to  teams,  and  studies  detailing 
iimovative  analytic  approaches  to  assessment  of  performance  were  reviewed.  This  body  of 
literature  provided  the  researchers  with  new  insights  which  were  implemented  as  part  of  the 
design.  This  review  of  theoretical  and  applied  literature  provided  a  valuable  source  of  support  in 
both  the  design  and  development  phases  of  the  project  and  should  be  considered  an  integral  part 
of  any  FEA. 


Training  Program  Design 

The  SGT  program  was  designed  to  follow  the  standard  ISD  methodology,  with 
responsibility  for  products  resting  on  a  cooperative  effort  by  the  SME,  the  ISD  specialist  and  the 
media  specialist.  The  creation  of  a  set  of  special  worksheets,  designed  to  provide  a  format  for  the 
development  of  component  parts  and  to  guide  the  structure  of  each  exercise,  proved  to  be  an 
invaluable  tool  for  accomplishing  this  effort.  The  worksheets  enforced  discipline  of  team 
members,  requiring  them  to  follow  soimd  principles  of  adult  learning  in  the  application  of  the 
SGT  training  methodology  (and  providing  a  structure  within  which  elements  of  the  exercises 
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could  be  developed).  Future  developers  should  consider  providing  similar  tools  to  guide  the  ISD 
portion  of  their  development  efforts. 

The  SGT  team  concentrated  on  filling  a  training  need,  using  existing  technology  to  run  staff 
exercises.  This  approach  allowed  the  Team  to  combine  the  latest  research  on  team  training, 
group  dynamics,  and  adult  learning  with  the  use  of  off-the-shelf  technology  to  fulfill  the  staff 
training  requirement.  The  efficiency  of  this  technique  (in  terms  of  manpower  use  and 
development  cost)  was  recognized  and  should  be  pursued  in  future  projects. 

Exercise  Development 
Exercise  Preview 

A  key  concern  for  commanders  was  the  current  over-reliance  on  bulky,  paper-based 
materials.  The  SGT  TSPs  converted  paper-based,  0/C  delivered  information  to  multimedia 
based,  auto-instructional  material.  This  approach  provided  units  with  the  flexibility  and  ease  of 
operation  they  needed  to  simplify  their  training.  The  automated  preview  reduced  the  knowledge 
requirements  of  0/Cs.  It  also  implemented  the  principle  of  “immersion”  into  SGT  training, 
shortening  the  time  required  to  understand  the  training  scenario  while  still  ensuring  a  complete 
presentation  of  necessary  background  information  every  training  iteration.  A  key  lesson  learned 
is  that  immersion  is  facilitated  by  multimedia  previews  that  are  accmate,  concise,  colorful,  and 
mentally  stimulating. 


Staff  Section  Preparation 

The  design  and  development  process  for  instructional  media  is  described  in  various  sources 
(e.g.,  Dick  &  Carey,  1985;  Kemp,  1985).  This  process  involves  the  integration  of  three  different 
skills:  (a)  staff  subject  matter  expertise,  (b)  ISD  expertise  (with  experience  on  instructional 
development  in  the  media  being  used),  and  (c)  a  trained  media  specialist.  While  it  is  possible  for 
one  person  to  have  all  three  skills,  the  norm  is  for  at  least  two  people  and  usually  three  to  be 
involved  in  this  process.  The  SGT  project  used  three  individuals.  The  typical  development 
procedure  for  the  staff  section  preparation  is  for  the  exercise  author — ^who  was  also  the  SME — ^to 
provide  information  to  both  the  ISD  specialist  and  the  media  specialist.  The  ISD  specialist  then 
would  review  the  information  and  work  with  the  media  specialist  who  would  develop  the  product 
(e.g.,  the  staff  section  preparation  program  or  exercise  preview).  The  draft  product  would  be 
returned  to  the  exercise  author  who  would  review  the  material  and  explain  any  necessary  changes 
to  the  ISD  specialist  and  media  specialist.  The  changes  would  then  be  made  as  appropriate.  Not 
adhering  to  this  process  led  to  inefficiencies  -  and  provided  a  reminder  that  use  of  the  correct 
ISD  process  can  prevent  costly  mistakes.  Future  projects  that  include  development  of  computer- 
assisted  training  should  have  appropriately  experienced  personnel  who  are  familiar  with  the  ISD 
approach  to  training  development. 


Pre-Execution  Staff  Huddle 


Confirming  the  commander’s  intent  and  understanding  the  “big  picture”  form  an  important 
part  of  troop  leading  procedures,  especially  within  the  context  of  the  back  briefing  fi:om  the  staff. 
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The  staff  huddle  is  a  currently  practiced  (but  often  unrecorded)  tried  and  true  military  procedure. 
The  development  of  the  pre-execution  staff  huddle  T^  materials  provided  the  Team  with  the 
opportunity  to  examine  the  effectiveness  of  this  technique  in  creating  a  shared  mental  model.  It 
confirmed  that  this  important  procedure  has  value  within  simulation-based  training  systems  and 
should  be  included  in  future  efforts. 


Exercise  Execution 


Subject  matter  experts.  The  SGT  team  discovered  that  they  did  not  have  the  breadth  of 
expertise  necessary  to  fully  develop  BOS  material  for  all  the  staff  sections,  especially  the  FS  and 
intelligence  sections.  Trial  and  pilot  participants  in  these  staff  sections  assisted  by  making 
numerous  suggestions  for  improving  or  adding  message  traffic  to  better  perform  their  staff  role. 
This  enabled  the  Team  to  create  detailed  messages  that  more  closely  replicated  real-time 
situations.  A  key  lesson  learned  from  the  current  effort  is  the  importance  of  considering 
alternative  sources  of  support  to  accommodate  project  requirements.  This  is  a  strategy  likely  to 
become  even  more  commonplace  given  the  constraints  associated  with  the  Army’s  current 
training  resources. 

Staff  tools.  During  the  trial,  the  staff  officers  indicated  that  a  more  complete  set  of  internal 
staff  tools  should  have  been  prepared  with  the  OPORDs  for  use  by  the  FS,  EN,  and  intelligence 
sections.  Staff  sections  reported  that  the  standard  job  aids  should  be  more  detailed  and  focused 
on  section-specific  battle  tasks.  They  reported  that  the  synchronization  matrix  and  DST  tools 
were  geared  for  the  commander  and  maneuver  elements  (i.e.,  had  a  S3  slant).  Developing 
additional  staff  tools  for  the  intelligence,  FS,  and  EN  sections  (primarily  staff  estimates  used  for 
preliminary  OPORD  work)  would  greatly  increase  the  amount  of  material  needed  for  support  of 
an  exercise.  The  lesson  learned  here  is  that  the  staffs  may  not  have  fully  appreciated  the  tools 
available  to  them  from  the  SGT.  Rather  than  developing  additional  tools  which  may  prove  to  be 
a  training  detractor,  more  emphasis  should  be  placed  on  understanding  and  using  the  tools  that 
are  already  provided.  Furthermore,  positive  feedback  regarding  the  power  charts  used  for  the 
exercise  point  to  their  potential  utility  and  lead  to  the  recommendation  that  they  be  put  in 
electronic  format  so  that  they  can  be  managed,  tracked,  and  updated  on  the  personal  computer 
provided  at  each  workstation. 

Interactor  cell.  The  ability  to  replicate  all  external  nodes  that  a  brigade  staff  might  query 
during  an  exercise  was  an  anticipated  challenge,  possible  only  with  exercises  that  were  highly 
structured  and  provided  a  general  exercise  tactical  flow.  The  fact  that  the  SGT  is  computer 
driven  (providing  the  message  list  and  all  activities  including  movement,  engagement  times, 
locations,  and  results)  contributed  positively  to  maintaining  exercise  structure  and  supporting  the 
interactor  cell.  The  challenge  was  to  provide  all  information  the  interactor  cell  would  need  in  a 
maimer  they  could  use,  and  then  train  those  interactors  on  both  their  functions  and  the  use  of  the 
material.  However,  without  dedicated  BOS  SMEs,  the  SGT  team  could  not  fully  anticipate 
queries  or  provide  the  interactor  with  cell  all  of  the  full  guidance.  Once  again,  the  need  for 
SMEs  for  all  BOS  areas  in  the  project  was  highlighted. 
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Staff  Section  After  Action  Review 


The  self-assessment  technique  implemented  for  the  staff  section  AARs  was  similar  to 
techniques  recommended  by  Bailey  et  al.  (1995)  and  Smith- Jentsch  et  al.  (1996).  This  technique, 
both  in  SGT  and  in  the  shipboard  team  training  environment,  is  dependent  on  a  structured 
training  exercise.  The  positive  comments  received  during  the  pilot  and  trial  indicated  the  value 
of  this  approach.  Lessons  learned  from  the  current  effort  point  to  the  importance  of  linking  self- 
assessment  to  the  learning  objectives  in  order  to  improve  staff  performance.  Likert  scale  ratings 
combined  with  written  rationales  of  the  ratings  served  as  a  useful  mechanism  for  collecting  self- 
assessment  data  on  staff  processes. 

Exercise  After  Action  Review 

The  exercise  AAR  provided  staffs  the  opportunity  to  reexamine  internal  staff  processes  (i.e., 
SOPs)  while  the  commander  and/or  XO  reviewed  learning  objectives  relevant  to  their  particular 
staff  (as  opposed  to  an  0/C  facilitated  session  wherein  there  is  the  potential  to  severely  limit  the 
training  audience’s  input,  or  to  focus  the  discussion  on  topics  other  than  execution  of  the  learning 
objectives).  Unexpectedly,  the  SGT  exercises  were  used  as  a  device  during  the  pilot  test  to 
evaluate  a  unit’s  SOP.  (As  an  example,  the  pilot  unit  discovered  that  their  SOP  contained  no 
information  on  staff  huddles.)  The  pilot  audience  adopted  portions  of  the  provided  SOP  to 
include  in  their  own  unit  SOP.  This  illustrates  the  potential  applicability  of  SGT  components  in 
assisting  units  with  SOP  development.  The  potential  may  be  even  more  important  to  explore 
within  the  context  of  shared  mental  models  and  evolving  TTPs  for  Force  XXI. 

Automation  of  Measurement 

The  SGT  performance  assessment  design  required  the  automatic  collection  of  performance 
indicators  by  the  computer  data  logger  along  with  semi-automated  data  collected  by  the  0/Cs. 
The  WOO  and  staff  fingerprint  data  was  provided  to  staff  sections  upon  completion  of  the 
exercise.  With  this  information  they  assessed  their  own  performance  on  the  learning  objectives. 
The  automated  measures  in  conjunction  with  the  computer-driven  staff  section  performance 
assessment  permitted  the  SGT  system  to  provide  structured  performance  assessments  oriented  on 
learning  objectives,  without  the  use  of  0/Cs. 

The  semi-automated  performance  checklists  completed  by  the  0/Cs  using  PDAs  provided 
additional  performance  data.  This  data,  consolidated  with  the  automated  data  and  each 
individual  sections’  self-assessment  responses,  were  provided  to  the  commander.  With  this 
combination  of  quantitative  and  qualitative  information,  the  commander  could  derive  an  action 
plan  for  future  training.  This  was  done  within  30  minutes  of  the  conclusion  of  the  training, 
thereby  creating  an  assessment  package  that  was  truly  ‘take-home.’  The  application  of  this 
technological  approach  increased  information  objectivity  while  improving  timeliness  of 
feedback. 

The  Team  believes  that  future  training  projects  should  examine  automated  performance 
assessment  to  reduce  operating  overhead  and  provide  unbiased  data  upon  which  the  individual 
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trainees  may  assess  their  performance.  The  use  of  PDAs  effectively  reduced  reliance  on  0/Cs, 
simplified  the  exercise  observation  process,  and  increased  observational  accuracy.  The  PDAs 
contained  checklists  cued  to  specific  actions  and  times  within  the  exercise,  and  alerted  the 
observer  as  to  who  and  what  to  observe.  A  key  lesson  learned  is  that  automating  the  collection 
of  data  had  high  payoffs  on  a  number  of  fronts  (e.g.,  reduction  in  number  of  required  observers, 
increased  observational  objectivity)  and  is  an  approach  that  should  be  adopted  in  future  efforts. 

Quality  Control 

Quality  control  was  a  challenge  in  the  exercise  development  process.  There  is  seldom 
adequate  time  or  resources  available  for  “total”  quality  control.  However,  when  new  software  is 
developed  even  one  small  error  can  cause  a  system  crash.  The  number  of  component  part 
inconsistencies  in  SGT  products  was  minimized  once  the  Team  instituted  the  use  of  development 
worksheets  and  insisted  on  strict  adherence  to  a  structured  design  methodology. 

Quality  control  was  an  important  aspect  of  developing  the  exercise  driver  for  the  current 
effort,  the  flow  of  messages  (or  message  send  stream).  The  exercise  messages  sent  to  each  staff 
section  replicated  the  messages  they  receive  during  an  actual  mission  of  the  type  being  trained. 
The  messages  provided  information  from  the  division  (higher),  adjacent,  subordinate,  and 
supporting  units.  The  automatically  generated  messages  provided  the  training  audience  with  all 
the  information  they  should  have  needed  to  determine  the  actual  “ground  truth”  situation.  The 
quality  of  the  section  AAR  depended  on  the  accuracy  and  timeliness  of  the  messages  arriving  at 
the  staff  section’s  workstation.  For  this  to  occur,  a  rigorous,  time-consuming  comparison  of  the 
actual  message  send  stream  had  to  be  made  against  many  separate  components  of  the  training 
system.  Even  with  numerous  reviews  and  adjustments  prior  to  both  the  pilot  and  the  trial,  some 
errors  went  imdetected.  This  highlighted  the  essential  nature  of  the  quality  control  process;  every 
opportunity  for  review  must  be  taken  to  ensure  the  delivered  products  are  correct  and  to  eliminate 
even  subtle  discrepancies.  Time  and  resources  to  support  thorough  quality  control  should  be  a 
high  priority  of  any  effort  and  will  provide  valuable,  long-term  benefits. 

Support  Requirements 

During  the  needs  assessment,  interviews  with  the  commanders  established  that  they  would 
not  have  the  training  staff  available  to  support  simulation-based  structured  training  as  previously 
designed  in  SGT  or  in  COBRAS.  The  previous  SGT  project  used  eight  individuals  for  the 
evaluation  of  a  brigade  exercise  (one  exercise  administrator,  two  interactors,  one  0/C  for  the  XO 
and  BC,  and  one  0/C  for  each  staff  section  evaluated  in  the  Main  CP).  With  COBRAS,  a  scaled- 
down  brigade-level  exercise  (evaluating  only  the  EN,  FS,  S2,  and  S3  sections)  using  BBS,  0/Cs, 
and  interactors  would  require  at  least  30  individuals  (Campbell,  Graves,  Deter,  &  Quinkert, 

1998). 

The  SGT  pilot  and  trial  were  conducted  using  only  four  trainers.  Two  0/Cs  observed  the 
staff:  the  senior  0/C  observed  the  S3  section,  BC,  and  XO;  the  junior  0/C  observed  the  FSE,  S2, 
and  EN  sections.  The  reduction  in  personnel  was  made  possible  because  interfaces  for  the 
exercise  administrator  and  interactors  were  improved  and  simplified,  and  the  0/Cs  were  provided 
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semi-automated  data  collection  devices.  The  devices  had  signals  to  alert  them  to  the  actions  they 
were  to  observe,  and  assist  them  in  keeping  their  data  observations  synchronized  with  the 
exercise  flow.  Potential  coaching  questions  were  also  provided  on  the  data  collection  devices, 
reducing  the  background  knowledge  requirements  of  the  0/Cs.  Again,  an  important  lesson 
learned  surrounds  the  judicious  use  of  technology  to  aid  inexperienced  or  short-handed  0/Cs.  It 
may  reduce  the  train-up  time  and  provide  a  level  of  consistency  across  performance  evaluations. 
In  addition,  it  may  also  reduce  the  number  of  0/Cs  required  for  an  exercise,  making  the  use  of  a 
particular  training  program  more  feasible  for  commanders. 

Project  Evaluation 
Formative  Evaluation  Time  Line 


During  the  current  effort,  the  Team  generally  followed  the  formative  evaluation  methods  as 
planned.  Certain  portions  were  delayed  or  moved  into  later  phases  of  development  because 
technology  innovations  could  not  always  be  implemented  as  programmed.  The  portion  of  formal 
evaluation  process  requiring  support  external  to  the  project  staff  was  conducted  late  in  the 
project.  Delaying  portions  of  the  evaluation  decreases  the  likelihood  of  incorporating  the 
findings  into  the  development  effort  and  jeopardizes  the  contract  delivery  schedule.  Thus,  the 
evaluation  should  be  conducted  and  revision  recommendations  provided  early  in  the 
development  cycle  so  that  they  can  be  used.  Formative  evaluation  events  should  be  spaced  to 
permit  time  to  implement  revisions.  However,  if  the  time  line  does  not  permit  revisions,  sound 
formative  evaluation  records  should  be  kept  for  use  in  any  follow-on  work 

Formative  Evaluation  Methods 


User  Panel 


The  user  panel  was  designed  to  be  composed  of  active  duty  military  members,  representing 
the  major  BOSs  of  the  brigade  staff  and  the  commander;  however,  only  one  position  (the 
command  representative)  was  filled  with  an  active  duty  soldier.  (Some  part  time  assistance  by 
another  active  military  member  was  provided.)  The  BOS  areas  were  reviewed  by  former  military 
and  reserve  component  members  not  part  of  the  SGT  team.  The  lack  of  active  duty  military 
assistance  originally  caused  concern.  Fortimately,  the  reviewers  brought  a  wealth  of  knowledge 
(both  tactical  and  training)  to  the  project.  While  the  Team  would  have  preferred  experienced 
experts  from  the  active  component  as  reviewers,  the  current  effort  demonstrated  the  utility  of 
pursuing  alternate  sources  of  support.  The  Team  also  learned  that  coverage  of  each  BOS  area  on 
the  user  panel  was  essential  to  ensuring  a  thorough  accounting  of  all  staff  activities  and 
requirements. 

Hot  Washes 

As  a  part  of  the  exercise  evaluation  process,  participants  in  the  pilot  and  trial  were  divided 
into  three  groups  for  the  hot  washes:  (a)  enlisted,  (b)  officer  (not  command  group),  and  (c) 
command  group.  This  separation  proved  to  be  beneficial  as  evidenced  by  the  number  and 
frankness  of  the  comments  that  were  received.  The  participants  appeared  to  be  comfortable  in 
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providing  detailed  feedback  on  all  aspects  of  the  exercises,  although  the  Team  had  to  separate 
negative  comments  fiieled  by  equipment  unfamiliarity  or  resistance  to  changes  in  training 
approach  from  constructive  feedback  directed  at  the  current  effort. 

Training  Versus  Data  Collection 

Not  enough  pilot/trial  iterations  were  run  to  allow  unit  persoimel  to  assume  the  observer 
roles.  This  was  especially  true  for  observer  materials  because  these  were  not  incorporated  from 
prior  work  as  were  messages  and  exercises.  Additional  excursions  and  repetitions  of  the 
exercises  (as  a  set)  will  be  needed  to  capture  a  more  complete  picture  of  the  efficacy  of  the 
materials;  T^  materials  and  touch  screen  interface  require  many  detailed  reviews  to  ensure 
verification  of  their  utility,  appropriateness,  and  effectiveness  for  the  training  audience. 

The  resourcing  challenges  in  the  Army  today  make  it  difficult  to  find  units  with  the  time  and 
money  needed  to  take  part  in  a  comprehensive  evaluation  of  a  new  training  system.  The  SGT 
team  sought  to  provide  the  training  audience  with  a  good  training  experience  while 
simultaneously  evaluating  the  system  and  gathering  insights.  It  is  difficult  to  evaluate  a  system 
and  provide  unit  training  at  the  same  time.  Still,  developers  must  be  flexible  and  anticipate 
unexpected  unit  training  requirements.  For  instance,  lack  of  unit  preparation  time  at  homestation 
may  require  trainers  to  provide  some  additional  instruction  during  the  WST  rather  than  allowing 
the  individuals  to  proceed  totally  self-paced. 

Participants 

The  participants  for  the  pilot  and  trial  were  from  intact  brigade  staffs,  representing  the  target 
audience.  These  units  were  both  from  the  ARNG.  The  fact  that  no  active  units  were  evaluated 
means  that  the  system  may  need  minor  modifications  to  fit  this  training  audience.  While  it  is 
important  that  pilot  and  trial  participants  closely  represent  the  audience  for  which  the  training  is 
designed,  it  is  also  important  that  developers  seek  alternative  sources  of  support  when  necessary. 
The  ARNG  provided  a  valuable  alternative  for  the  current  effort. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


General 

The  current  effort  demonstrated  that  team  process  training  can  be  implemented  in  a 
structured  computer-driven  learning  environment.  Development  of  such  training  requires  a 
multi-disciplinary  organization  of  SMEs,  ISDs,  formative  evaluation  experts,  media  designers, 
research  psychologists,  technology  specialists,  and  software  engineers.  This  team  built  a  training 
framework  to  fit  the  user’s  needs  with  recognized  small  group  training  concepts,  adult  learning 
methodologies,  and  their  understanding  of  the  training  requirements. 

This  TSP  was  designed  to  meet  the  brigade  staff  training  needs  for  a  system  which  would 
bridge  the  gap  between  individual  training  and  interactive  simulation  based  training.  Interactive 
simulation  training  traditionally  has  high  overhead.  The  Army  needs  a  way  to  ensure  the  “most 
bang  for  the  buck.”  Although  not  measured  analytically,  it  would  appear  from  the  formative 
evaluation  insights  collected  during  this  project  that  this  TSP  correctly  fills  the  gap.  More  trials 
are  needed  to  verify  these  findings.  If  fielding  is  to  be  considered,  three  steps  are  recommended: 

1 .  The  training  packages  must  be  converted  to  run  on  computers  that  have  an  operating 

system  which  is  commercially  available  (e.g.,  Windows  95  or  NT). 

2.  The  complete  exercise  library  should  be  developed  in  order  to  provide  robust  training. 

3.  All  the  T^  materials  should  be  computer  driven  packages. 

In  light  of  the  overhead  constraints  normally  placed  upon  military  training  programs,  this 
can  only  be  implemented  if  teehnology  innovations  are  applied  wherever  possible.  The  SGT 
team  did  not  attempt  to  fit  the  training  to  the  technology,  as  frequently  occurs.  The  probability  of 
successfully  implementing  a  training  program  that  relies  on  technology  is  greatly  improved  by 
ensuring  that  the  technology  and  software  personnel  clearly  understand  the  training  objectives. 

The  SGT  team  implemented  many  technology  innovations.  These  were  not  done 
haphazardly.  The  Team  examined  requirements,  constraints,  and  limitations.  Potential 
approaches  were  linked  to  each  requirement  and  the  most  cost  effective  approach  which  would 
satisfy  the  training  requirement  was  selected.  The  contributions  of  the  technology  innovations 
are  outlined  in  Appendix  D.  It  is  the  opinion  of  the  Team  that  the  most  important  innovations 
were  those  which  contributed  to  timely,  process-oriented  feedback  and  remediation  and  to  low 
overhead.  These  training  innovations  were  accomplished  by  using:  (a)  a  computer-driven 
struetured  training  delivery  means,  (b)  an  instrumentation  package  designed  to  collect 
performance  information  directly  related  to  the  learning  objectives,  (c)  a  time  hacked,  observer 
checklist  linked  to  the  learning  objectives  and  running  on  an  easy  to  operate  PDA,  (d) 
multimedia  AARs  covering  each  learning  objective,  and  (e)  highly  structured  T^  packages. 

As  a  result,  a  program  like  SGT  could  provide  a  staff  training  system  with  low  overhead, 
quick,  relevant  feedback  on  performance,  a  training  program  which  complements  eommanders’ 
training  goals,  and  an  AAR  cycle  that  is  not  totally  dependent  upon  trained  observers. 
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Recommendations  for  Further  Research 

The  following  list  is  based  on  this  project’s  lessons  learned  and  highlights  areas  for  further 
research  and  development  to  meet  unit  commanders’  needs. 

1 .  Additional  work  should  investigate  the  theoretical  concepts  used  in  this  project, 
especially  training  on  the  “shared  mental  model”  construct. 

2.  Further  work  is  needed  to  ensure  that  the  TSP  meets  active  duty  training  requirements. 

3 .  The  implementation  of  computer-generated  audio  cues  needs  to  be  tested  to  see  if  this 
contributes  to  further  staff  improvements  in  performance. 

4.  Further  work  is  needed  to  enhance  the  configuration  and  presentation  of  TSPs  based  on 
current  multimedia  and  ISD  research. 

5.  Investigations  should  be  conducted  on  whether  the  additional  exercises  are  required 
to  fully  prepare  a  staff  for  Janus  or  BBS. 

6.  Further  work  is  required  to  investigate  the  connections  between  individual  and  staff 
group  training  programs,  as  well  as  between  staff  training  programs  and  whole  unit 
training  systems. 

7.  Consider  adapting  SGT  for  current  Army  Tactical  Command  and  Control  Systems 
(ATCCS)  application 

The  SGT  project  has  pushed  forward  the  frontiers  of  team  process  training.  This  work 
provides  a  solid,  theoretically  grounded  foundation  for  future  research.  More  research  is  needed 
to  investigate  the  working  of  staff  groups  and  the  application  of  new  technology  to  staff  training 
needs. 
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APPENDIX  A 

ACRONYMS  AND  ABBREVIATIONS 


AAR 

ADA 

AR 

ARI 

ARNG 

ARTEP 

ATCCS 

BBS 

BC 

BDA 

BOS 

BSTS 

C/ST 

C' 

CAS^ 

CBI 

CCIR 

CGSC 

COA 

COBRAS 

CP 

CTC 


After  Action  Review 
Air  Defense  Artillery 
Armor 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

Army  National  Guard 

Army  Training  and  Evaluation  Program 

Army  Tactical  Command  and  Control  System 

Brigade/Battalion  Battle  Simulation 

Battle  Captain 

Battle  Damage  Assessment 

Battlefield  Operating  System 

Battle  Staff  Training  System 

Commander/Staff  Trainer  (currently  called  Staff  Group  Trainer) 

Command  and  Control 

Combined  Arms  and  Services  Staff  School 

Computer-based  Instruction 

Commander’s  Critical  Information  Requirement 

Command  and  General  Staff  College 

Course  of  Action 

Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved 
through  Simulation 

Conunand  Post 

Combat  Training  Center 
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cvcc 

DST 

EN 

ENDEX 

FA 

FC 

FEA 

FRAGO 

FS 

FSE 

HACC 

HQ 

IN 

ISD 

ITTBBST 

LAN 

LTC 

MAJ 

METT-T 

MI 

MSG 

MTC 

MTP 

MUTA 


Combat  Vehicle  Command  and  Control 

Decision  Support  Template 

Engineer 

End  of  Exercise 

Field  Artillery 

Field  Circular 

Front  End  Analysis 

Fragmentary  Order 

Fire  Support 

Fire  Support  Element 

Higher  and  Adjacent  Command  and  Control 

Headquarters 

Infantry 

Instructional  Systems  Design 

Innovative  Tools  and  Techniques  for  Brigade  and  Below  Staff 
Training 

Local  Area  Network 
Lieutenant  Colonel 
Major 

Mission,  Enemy,  Terrain,  Troops  and  Time  Available 

Military  Intelligence 

Master  Sergeant 

Movement  to  Contact 

Mission  Training  Plan 

Multiple  Unit  Training  Assembly 
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NCO 

Non-Commissioned  Officer 

NTC 

National  Training  Center 

0/C 

Observer/Controller 

OPORD 

Operation  Order 

PDA 

Personal  Digital  Assistant 

POI 

Program  of  Instruction 

R&S 

Reconnaissance  and  Surveillance 

RFI 

Request  for  Information 

SI 

Personnel 

S2 

Intelligence 

S3 

Operations 

S4 

Logistics 

SACC 

Subordinate  and  Adjacent  Command  and  Control 

SGM 

Sergeant  Major 

SGT 

Staff  Group  Trainer 

SIMBART 

Simulation-based  Mounted  Brigade  Training  Program 

SIMNET 

Simulation  Networking 

SIMUTA 

Simulation-based  Multiechelon  Training  Program  for  Armor  Units 

SitMap 

Situation  Map 

SME 

Subject  Matter  Expert 

SOP 

Standing  Operating  Procedure 

STARTEX 

Start  of  Exercise 

73 

Train-the-Trainer 

TAC  CP 

Tactical  Command  Post 
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THP 

Take  Home  Package 

TOC 

Tactical  Operations  Center 

TRADOC 

U.S.  Army  Training  and  Doctrine  Command 

TSP 

Training  Support  Package 

TTP 

Tactics,  Techniques,  and  Procedures 

WOO 

Window  of  Opportunity 

WST 

Workstation  operator  training 

XO 

Executive  Officer 
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APPENDIX  B 

THEORETICAL  CONSTRUCTS  UNDERLYING  THE  STAFF  GROUP  TRAINER 


Much  of  the  foundation  of  the  SGT  project  is  based  on  a  study  of  teams  and  how  they 
accomplish  their  tasks.  The  application  of  this  research  to  the  current  effort  is  discussed  below. 

Teams 

A  team  consists  of  “two  or  more  people  with  different  tasks  who  work  together  adaptively  to 
achieve  specified  and  shared  goals”  (Brannick  &  Prince,  1997,  p.  4).  These  authors  state  that 
teams  have  the  following  characteristics: 

1 .  The  team  itself  is  not  an  ad  hoc  formation. 

2.  The  team  members  generally  have  distinct  functions. 

3.  The  team  tasks  require  simultaneity  (multiple  team  members  on  same  task)  and 
sequencing  (one  team  member’s  output  is  input  to  another  team  member),  or  both. 

This  information,  along  with  the  discussion  of  the  interdependence  of  individuals  on  teams 
by  Convert,  Craiger,  and  Cannon-Bowers  (1995),  was  used  by  the  Team  to  identify  the  brigade 
staff  as  a  representation  of  a  team.  Those  functions  performed  by  the  brigade  staff  (and  by  the 
individual  staff  sections)  were  examined  in  conjunction  with  the  development  of  the  SGT  team 
processes. 

Team  Processes 

A  team  process  refers  to  “the  behavioral,  procedural,  and  temporal  phenomena  that  describe 
a  team’s  functioning”  (Convert  et  al.,  1995,  p.  151).  These  processes  are  dynamic  and  evolve 
over  time.  Training  designers  and  developers  must  recognize  and  rmderstand  these  processes  in 
order  to  correctly  design  training  which  will  train  teams  to  perform  effectively  (Coo vert  et  al., 
1995).  The  processes  must  be  identified  for  each  training  environment. 

On  the  battlefield,  the  job  or  function  of  a  military  staff  is  to  support  the  commander’s 
decision  making  process.  This  decision  making  process  was  established  in  the  previous  SGT 
effort  as  the  U.S.  Army’s  commander’s  C^  cycle  with  the  sequential  components  of  See,  Assess, 
Decide,  and  Act  (Department  of  the  Army,  1995b).  The  staff  procedures  and  behaviors  that 
support  this  function  are  the  foundation  upon  which  this  staff  training  system  is  designed.  Each 
of  these  staff  team  processing  components  supports  the  commander’s  actions  in  some  manner 
(ARI,  1997). 

Research  on  team  decision  making  in  a  military  environment  previously  identified  the 
behaviors  of  gathering,  processing,  integrating,  and  communicating  information  as  important 
processes  to  support  task-relevant  decisions  (Cannon-Bowers,  Salas,  &  Converse,  1993).  These 
researchers  identified  that  team  decision  making  required  team  members  to  process  and  filter  raw 
data,  apply  individual  expertise,  communicate  relevant  information,  and  (often)  make 
recommendations  to  other  members.  In  a  hierarchically  structured  team,  such  as  a  military  staff, 
where  final  decision  authority  is  retained  by  a  single  individual,  the  staff  provides  the  decision 
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maker  with  assessments  and  mformation  that  are  crucial  to  the  situation  (Cannon-Bowers  et  al., 
1993).  These  behaviors  were  developed  into  a  set  of  learning  objectives  for  the  previous  SGT 
effort — ^monitor,  process,  analyze/evaluate,  communicate,  coordinate,  integrate,  recommend, 
disseminate,  synchronize,  and  direct — and  further  refined  for  use  in  the  current  SGT. 

Team  Strategies  and  Teamwork 

Team  strategies  consist  of  “a  set  of  behaviors  and  interaction  patterns  initiated  by  the  team  in 
response  to  situational  demands”  (Convert  et  al.,  1995,  p.  152).  For  the  SGT  project,  the 
situational  demands  were  created  by  and  linked  to  tactical  scenarios  for  the  training  exercises. 
This  aligned  the  learning  objectives  of  the  exercises  with  the  flow  of  the  battle,  creating  the 
opportunity  for  successful  performance  of  staff  tasks.  Teamwork  has  been  defined  as  “the  set  of 
behaviors  executed  by  two  or  more  individuals  as  a  function  of  coordinating  requirements 
imposed  by  interdependent  tasks  in  achieving  common  goals”  (Brannick,  Prince,  Prince,  & 

Salas,  1995,  p.  642).  Cannon-Bowers  et  al.  (1993)  saw  teamwork  skills  as  those  skills  which 
make  individuals  effective  team  members.  The  work  of  Cannon-Bowers  et  al.  (1993)  was  very 
important  to  the  current  effort;  the  authors  stressed  the  need  for  a  highly  stmctured  training 
design  implementation  in  order  to  build  team  effectiveness. 

Shared  Mental  Models 

The  term  “shared  mental  model”  has  been  used  for  the  organized  knowledge  shared  by  team 
members  who  work  together  over  relatively  long  periods  of  time  (Orasanu  &  Salas,  1993). 
Research  has  indicated  that  for  a  team  to  work  well  together,  the  team  members  must  “develop  a 
common  understanding  of  the  problem  and  a  strategy  for  solving  it.  This  includes  an  agreement 
about  each  member’s  role  in  the  team  and  expectations  about  behavior”  (Morgan  &  Bowers, 
1995,  p.  282).  “Such  knowledge  enables  each  person  to  carry  out  his  or  her  role  in  a  timely  and 
coordinated  fashion,  helping  the  team  to  function  as  a  single  unit  with  little  negotiation  of  what 
to  do  and  when  to  do  it”  (Orasanu  &  Salas,  1993,  p.  7).  U.S.  Army  doctrine  has  also  established 
a  basis  for  a  staffs  “shared  mental  model”  (Department  of  the  Army,  1988a;  1988b;  1988c; 
1988d;  1992).  Effective  training  must  provide  a  method  for  team  members  to  construct  the 
common  and  necessary  mental  model  of  the  problem  circumstances  (Rouse  &  Morris,  1986),  and 
provide  the  cues  for  each  team  member  which  will  lead  to  common  interpretations  of  the  events 
which  take  place  (Cannon-Bowers,  Salas,  &  Converse,  1993).  For  battle  staff  personnel  this 
translates  into  a  shared  understanding  of  the  events  on  the  battlefield  as  they  occur  and  a 
common  placement  of  them  into  the  context  of  the  overall  tactical  mission. 

The  Team  designed  the  training  project  so  that  leaders  would  specify  the  mental  model  to  be 
shared.  Research  has  shown  that  leaders  of  high-performing  teams  stated  more  plans,  considered 
more  options,  provided  more  explanations,  and  sounded  more  warnings  or  predictions  than  low 
performing  teams  (Orasanu,  1990).  The  SGT  training  project’s  design  concept  was  built  on  this 
research  background  and  Army  doctrine,  and  it  formulated  conditions  to  guide  the  staff  in 
developing  a  shared  mental  model.  The  Team  also  established  learning  objectives  that 
emphasized  process-related  goals  (Jourdan,  Bandura,  &  Banfield,  1991). 
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Adult  Learning 


Research  in  adult  learning  and  staff  training  has  indicated  that  learning  is  most  effective  if 
the  instruction  is  designed  to  engender  successful  accomplishment  as  a  part  of  the  learning 
process  (Jourdan  et  al.,  1991).  When  first  teaching  a  complex  task,  a  learner’s  self-confidence 
beliefs  and  perception  of  success  were  enhanced  by  emphasizing  process-related  (learning)  goals 
over  outcome-related  (performance)  goals.  Success  was  redefined  to  include  effort,  form,  and 
strategy,  rather  than  winning  or  losing  or  counting  the  number  of  tasks  completed  (Jourdan  et  al., 
1991).  Olmstead  (1992)  pointed  out  that  other  researchers  (Mills,  1967;  Gill,  1977)  found  that 
nothing  contributed  more  to  improved  cohesiveness  than  a  successful  action.  Druckman  and 
Bjork  (1994)  wrote  that  by  ensuring  successful  outcomes,  individual  self-confidence  is  built  and 
this  self-confidence  is  a  potent  predictor  of  fiiture  successful  performance.  Thus,  one  goal  of  the 
training  project  design  was  for  the  staff  to  be  challenged  but  be  able  to  achieve  each  exercise’s 
training  objectives  successfully. 


Training  Design  Philosophy 

According  to  McIntyre  and  Salas  (1995); 

Tactical  teams  within  the  military  exist  to  (1)  to  help  a  leader  assess  a  given  scenario 
involving  imminent  danger  or  threat,  (2)  to  provide  information  to  the  leader  in  a  form 
that  he  or  she  can  use  in  making  a  decision,  and  (3)  to  implement  the  action  implied  by 
the  decision  that  the  leader  comes  to.  (p.  9) 

This  requires  a  trained  staff.  The  staff  members  must  be  able  to  individually  make  decisions 
by  processing  system-specific  knowledge  as  well  as  function  as  members  of  a  team  sharing 
individual  knowledge  and  conclusions.  They  then  must  process  pooled  information  into  system- 
wide  decisions  (Druckman  8c  Bjork,  1994).  When  taken  in  the  context  of  training,  the 
commander  interacts  with  the  staff  as  the  “principal  instructor”  or  “head  trainer”,  establishing 
the  goals  and  basic  model  to  which  all  staff  members  must  contribute  their  knowledge,  expertise, 
and  effort. 

This  effort  is  not  the  first  to  use  these  training  precepts.  The  basic  theoretical  design  for  the 
research  conducted  in  the  current  effort  was  taken  from  the  Bailey  et  al.  (1995)  project  guidelines 
for  facilitating  team  training  which  included  concepts  for  the  development  of:  (a)  training 
scenarios  with  event-based  behavior  triggers,  (b)  team  performance  measures,  (c)  use  of  a  timely 
performance  feedback  system,  and  (d)  trainer  training  on  how  to  assess  teamwork  skills. 

For  their  project,  Bailey  et  al.  (1995)  developed  tactical  scenarios  which  had  trigger  events 
to  cue  individual  task  behaviors,  as  well  as  teamwork  behaviors.  In  addition,  other  investigators 
such  as  Hall  et  al.  (1993)  state  that  it  is  very  important  to  use  highly  structured  scenarios  which 
have  been  specifically  designed  to  elicit  appropriate  decision  making  actions  or  critical  team 
actions. 

Bailey  et  al.  (1995)  also  developed  a  measurement  device  which  was  used  to  capture 
information  on  situational  awareness,  communication,  compensatory  behavior,  and  team 
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leadership.  The  performance  feedback  materials  consisted  of  integrated,  automatically  collected 
machine  data,  0/C  data  on  effective  and  ineffective  teamwork  behaviors,  and  0/C  ratings  of 
team  performance  on  a  Likert  scale  with  behavioral  anchors.  These  dynamics  were  captured 
along  the  scenario  event  time  line  and  debriefed  within  15  minutes  after  the  exercise  concluded. 

Bailey  et  al.  (1995)  went  on  to  train  the  exercise  observers  on  how  to  assess  teamwork  skills 
using  a  video  tape  of  a  previous  exercise.  They  offered  the  following  guidelines  for  designing 
rater  (observer)  training: 

1 .  Familiarize  raters  with  the  critical  trigger  events  within  each  scenario. 

2.  Provide  information  regarding  scenario  trigger  events,  and  target  task  skills  and  teamwork 
behaviors. 

3.  Observe  teams  demonstrating  effective  and  ineffective  teamwork. 

4.  Practice  identifying  target  behaviors  and  assigning  ratings. 

5.  Provide  feedback  on  the  accuracy  of  the  rater’s  assessments. 

Debriefings  were  structured  around  the  trigger  events  which  facilitated  preparation  and 
helped  focus  on  specific  behaviors  and  actions.  Bailey  et  al.  (1995)  found  that  developing  a 
debriefing  guide  helped  teams  identify:  (a)  their  strengths,  (b)  areas  of  opportunity  for 
improvements,  and  (c)  team  goals  to  improve  performance.  Bailey  et  al.  (1995)  also  found  that 
soliciting  team  input  as  to  why  certain  actions  were  taken  or  omitted  helped  to  establish  an 
interactive  dialogue  between  team  members  and  the  debriefer. 
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Workstation  Operator  Training  Questionnaire  C-2 

Exercise  Evaluation  Focus  Group  Interview  Questions  C-4 
Command  Post  Exercise  Questionnaire  C- 1 6 


C-1 


Name: 


Workstation  Operator  Training 

Instructions:  For  the  following  questions,  fill  in  the  circle  under  the  category  that  most  closely 
matches  your  response. 


1 .  How  effective  was  the 

workstation  operator  training  in 
getting  you  ready  to  be  a 
workstation  operator? 

-3 

Extremely 

Ineffective 

0 

-2 

0 

-1 

0 

0 

Neutral 

0 

1 

0 

+2 

0 

+3 

Extremely 

Effective 

0 

2.  Based  on  the  workstation 
operator  training,  how  sure  are 
you  that  you  could: 

-3 

Extremely 

Unsure 

-2 

0 

Neutral 

I 

+2 

+3 

Extremely 

Sure 

create  and  post  a  map  symbol 
to  your  section’s  map  display  ? 

0 

0 

0 

0 

o 

0 

0 

post  the  appropriate  overlays  to 
your  section’s  map  display? 

0 

0 

0 

0 

0 

0 

0 

post  an  overlay  to  the  CP’s 
Situation  Display? 

0 

0 

0 

0 

0 

0 

0 

create  messages  using  the 
appropriate  message  format? 

o 

0 

0 

0 

o 

0 

0 

send  a  message  to  the  correct 
addressee? 

0 

0 

0 

0 

0 

0 

0 

3.  In  your  opinion,  in  order  to 

0 

Olo  1/2 

1/2  to  1 

1  to  1  1/2 

1  1/2  to  2 

2  to  3 

More  than 

perform  adequately  as  a 
workstation  operator  how  much 

None 

hour 

hours 

hours 

hours 

hours 

3  hours 

additional  practice  would  you 
require? 

0 

0 

0 

0 

o 

0 

0 

4.  What  could  be  done  to  improve  the  Workstation  Operator  Training?  Comments? 

Contim/ed  on  next  page 
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Name: 


Workstation  Operator  Training,  Continued 


What  were  the  three  strongest  features  of  the  Workstation  Operator  Training? 

1. 

2. 

3. 

□  There  were  no  strong  features. 

What  were  the  three  weakest  features  of  the  Workstation  Operator  Training? 

1. 

2. 

3. 

Q  There  were  no  weak  features. 

Was  there  anything  that  must  be  changed  before  this  Workstation  Operator  Training  is  useable? 

1. 

2. 

3. 

□  There  is  nothing  that  needs  to  be  changed  for  the  Workstation  Operator  Training  to  be 
useable. 
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Exercise  Evaluation  Focus  Group 


Focus  Group 
Organization 


Ground  work 


Exercise 

Questions 


•  Focus  groups  are  organized  to  attempt  to  bring  out  as  much  information  as 
possible  in  the  short  time  provided. 

•  There  are  four  focus  as  follows: 

=>  Command  Group 

=>  Staff  Officers 

=>  StaffNCOs 

•  Each  group  is  basically  a  peer  group.  This  is  done  to  encourage  everyone  to 
speak  their  mind  and  not  look  to  the  senior  person. 

•  Each  group  has  a  slightly  different  perspective  based  on  their  roles  in  the 
training. 


•  Ensure  that  everyone  in  the  group  knows  the  role  the  others  in  the  group 
played  in  the  last  exercise. 

•  Ensure  the  group  understands  that  your  questions  are  focused  on  them-not 
the  commander  or  anyone  else.  We  are  interested  in  what  each  member  of 
the  group  has  to  say  and  his  individual  opinion. 

•  Our  focus  is  on  making  this  training  better  for  each  and  every  participant. 
Each  of  the  participants  entered  the  exercise  with  different  levels  of  training 
and  experience.  Each  of  them  had  a  different  experience  in  the  exercise  and 
the  training  program.  Therefore,  each  individual’s  views  are  coming  from  a 
slightly  different  experience  and  each  is  important. 


1 .  What  did  you  like  best  about  the  exercise? 

(What  has  been  most  helpful  to  you?) 

2.  What  did  you  like  the  least  about  the  exercise? 

(What  was  least  helpful  to  you?) 

3.  What  should  be  changed? 

4.  What  should  be  continued  just  as  it  is  now? 

5.  What  should  be  continued  but  fine  tuned? 

6.  What  should  be  dropped? 

7.  What  would  it  take  to  make  this  exercise  more  beneficial  to  you? 

8.  Do  you  have  any  other  advice  about  the  exercise? 
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Specific  Questions  for  Each  Segment  of  the  Exercise 


Questions  •  The  following  are  questions  focused  on  each  segment  of  an  exercise. 

•  These  questions  should  be  used  in  the  final  training  program  focus  group. 

Exercise  •  What  did  you  think  about  the  exercise  preview? 

Preview  •  The  purpose  of  the  preview  was  to  quickly  get  you  into  the  exercise.  How 

effective  was  the  preview  in  doing  this? 

•  What  helped  you  the  most?  What  helped  you  the  least?  Should  anything  be 
changed?  Should  anything  be  added?  Dropped? 

Staff  Section  •  Are  there  any  new  T&Ps  that  we  are  not  using? 

Exercise  •  Were  the  power  chart  symbologies  understandable/acceptable? 

Preparation  ,  Was  the  material  written  at  the  correct  level? 

•  Were  expected  staff  actions  clear  to  you? 

•  Did  the  self-assessment  questions  focus  on  what  you  needed  to  know  to 
conduct  the  exercise? 

Mini-Rehearsal  •  How  long  should  the  mini-rehearsal  be? 

•  Should  we  put  structure  into  the  mini-rehearsal  or  leave  allow  more 
freedom  to  the  XO/BC? 


Execution  •  How  was  the  message  traffic?  Were  there  enough  messages?  Was  the 

message  traffic  about  what  you  would  expect  your  section  to  receive? 

•  Was  your  section  able  to  keep  the  power  charts  current?  Were  there  any 
difficulties?  Was  the  message  traffic  sufficient  for  you  to  keep  the  power 
charts  up  to  date? 

•  Was  the  interactor  cell  responsive  to  you? 

•  Besides  voice  and  digital  radio,  and  fax  traffic,  is  there  a  feature  that  needs 
to  be  emulated  that  is  not  there  now? 

•  How  did  you  section  function  as  compared  to  normal  operations?  Were  the 
interactions  similar  or  different?  Did  your  section  team  anything  about  its 
interactions  and  how  to  improve  them?  Could  you  transfer  what  you  learned 
about  your  interactions  to  normal  operations? 

•  Were  you  coached  during  the  exercise?  Was  this  coaching  effective?  Does 
the  coaching  needed  to  be  changed  or  modified? 

Continued  on  next  page 
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Specific  Questions  for  Each  Segment  of  the  Exercise,  Continued 


Staff  Section 
AAR 


•  What  did  you  think  of  the  Section  AAR? 

•  What  were  its  strengths  and  weaknesses? 

•  Should  anything  be  changed  or  refined  in  it? 

•  How  effective  was  your  Section  AAR? 

•  Was  the  discussion  focused  on  processes? 

•  Did  the  self-correction  sessions  focus  on  processes  or  outcomes? 

•  How  good  was  your  action  plan? 

•  Were  the  instructions  and  tools  adequate  for  your  section  to  conduct  its  own 
section  AAR? 


Large  Group 
AAR 


•  What  did  you  think  of  the  Large  Group  AAR? 

•  What  were  its  strengths  and  weaknesses? 

•  Should  anything  be  changed  or  refined  in  it? 

•  How  effective  was  the  Large  Group  AAR? 

•  Was  the  discussion  focused  on  processes  and  how  to  improve  those 
processes? 

•  Did  the  self-correction  sessions  focus  on  processes  or  outcomes? 

•  Were  the  right  things  discussed  in  the  Large  Group  AAR? 

•  How  good  was  your  action  plan? 

•  Do  you  think  your  unit  could  have  conducted  an  effective  Large  Group 
AAR  with  the  tools  available? 
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Summary 


Guides 


1 .  Before  the  focus  group,  be  sure  you  know  the  key  questions  and  the 
approximate  time  the  moderator  plans  to  spend  on  each  key  question. 

2.  Be  clear  in  your  mind  about  the  purpose  of  the  focus  group.  The  summary 
should  tie  closely  to  this  purpose. 

3.  Take  notes  with  two  things  in  mind:  first,  notes  that  will  help  you  provide 
a  brief  oral  summary  and,  second,  notes  for  your  detailed  analysis  after  the 
focus  group. 

4.  Begin  your  oral  summary  with  the  most  important  findings,  regardless  of 
when  they  were  discussed  in  the  focus  group.  Don’t  worry  about  the 
question  sequence  when  you  construct  your  summary. 

5.  Begin  your  summary  with  findings— what  was  actually  said.  Attempt  to 
capture  common  themes  but  also  acknowledge  differing  points  of  view. 
This  descriptive  summary  repeats  what  was  said  but  is  very  brief.  After 
you’ve  given  the  summary  of  what  was  said,  consider  offering  an 
interpretation.  The  interpretive  summary  attaches  additional  meaning  and 
goes  beyond  the  actual  words. 

6.  Listen  for  what  was  not  said  but  might  have  been  expected.  If  these  areas 
are  important,  then  in  the  summary  you  might  say,  “Some  things  were  not 
mentioned  like . . .  and  I  am  assuming  they  are  not  important.”  Look  at  the 
participants  while  you’re  saying  this  and  watch  for  reactions. 

7.  Cite  key  phrases  used  in  the  discussion.  This  demonstrates  connectedness 
and  careful  listening. 

8.  Keep  the  summary  to  3  minutes  or  less.  If  you  ramble  on,  people  will  tune 
out. 

9.  When  finished,  look  at  the  participants  and  ask,  “Is  this  summary 
complete?”  or  “Does  this  sound  OK  to  you?” 
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Responses  to  Questions 


SGT2  Trial  (April  1998) 

Exercise  Evaluation  Focus  Group 


C-8 


PT#60-04 


Responses  to  Questions,  continued 
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Responses  to  Questions,  Continued 
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Responses  to  Questions,  continued 
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Continued  on  next  page 
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Responses  to  Questions,  Continued 


5.  What  should  be  continued  but  fine  tuned? 

Brief  Summary/Key  Points 

Notable  Quotes 

Comments/Observations 


Continued  on  next  page 
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Exercise  Evaluation  Focus  Group 


Responses  to  Questions,  Continued 


6.  Wh  at  sh  ould  be  dropped? 


Brief  Summary/Key  Points  Notable  Quotes 


Continued  on  next  page 
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Responses  to  Questions,  continued 


7.  What  would  it  take  to  make  this  exercise  more  beneficial  to  you? 

Brief  Summary/Key  Points  Notable  Quotes 
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Responses  to  Questions,  Continued 
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Last  Name:  _ 

Position:  _ 

Command  Post  (CP)  Exercise 


CP  Exercise  The  staff  sections  practice  their  intra-  and  inter-staff  section  communication 
Purpose  and  coordination  skills.  The  command  post  builds  upon  previous  experience 

to  improve  performance  and  teamwork  in  providing  the  commander  and 
subordinate  and  higher  headquarters  information  they  need. 

Role  of  the  The  commander  is  the  lead  trainer  for  the  Staff  Group  Trainer  Program  rather 
commander  than  a  trainee.  He  trains  his  staff  to: 

•  provide  him  information  for  his  decisions, 

•  help  him  make  his  decisions, 

•  help  him  implement  his  decisions,  and 

•  keep  subordinate  and  higher  headquarters  informed. 

This  program  is  not  designed  to  teach  the  staff  tactics. 

Questionnaire  This  questionnaire  is  organized  into  sections  corresponding  to  the 
Organization  organization  of  the  exercise.  There  are  sections  for  the  following  phases: 

•  Exercise  Preview 

•  Staff  Preparation 

•  Mini-Rehearsal 

•  Exercise  Execution 

•  Section  AAR  (SAAR) 

•  Staff  (Large  Group)  AAR 

There  is  also  a  section  at  the  end  for  you  to  evaluate  the  entire  exercise. 

Importance  of  Provide  designers  and  developers  with  information  needed  to  improve  the 
your  response  exercise. 

Instructions  •  For  the  following  questions,  fill  in  the  circle  for  the  response  that  most 
closely  matches  your  rating. 

•  Please  provide  your  comments  or  suggestions  in  the  comment  block  at  the 
end  of  each  section. 


SGT2  Trial  (April  1998) 
Command  Post  Exercise 


C-16 


PT#60-04 


Exercise  Preview 


1 .  How  effective  was  the  preview 
in  getting  the  staff  ready  to 
conduct  the  exercise? 

-3 

Extremely 

Ineffective 

-2 

-1 

0 

Neutral 

+  I 

+2 

+3 

Extremely 

Effective 

0 

0 

0 

0 

0 

0 

0 

2.  Did  you  have  a  clear  picture  of: 

-3 

Extremely 

Unclear 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Clear 

What  led  up  to  this  phase  of  the 
battle? 

0 

0 

0 

0 

0 

0 

0 

What  the  commander  expected 
to  happen  in  this  phase  of  the 
battle? 

0 

0 

0 

0 

o 

0 

0 

What  the  commander  expected 
from  the  staff  for  this  phase  of 
the  battle? 

0 

0 

0 

0 

0 

0 

0 

3.  How  appropriate  was  the  length 
of  the  Exercise  Preview? 

-3 

Far  Too 
Short 

-2 

-1 

0 

About 

Right 

+1 

+2 

+3 

Far  Too 
Long 

0 

0 

0 

0 

0 

0 

0 

4.  What  could  be  done  to  improve  the  Exercise  Preview?  Comments? 
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Section  Preparation 


1 .  How  effective  was  the  section 

-3 

-2 

-1 

0 

+  1 

+2 

+3 

preparation  in  getting  your 

Extremely 

Ineffective 

Effective 

section  ready  to  conduct  the 

exercise? 

0 

0 

0 

0 

0 

0 

0 

2.  Did  you  have  the  information 

-3 

-2 

-1 

0 

1 

+2 

+3 

(locations  and  status  of  units, 

Far  Too 
Little 

About 

Right 

Far  Too 
Much 

overlays,  current  battlefield 

situation,  etc.)  you  needed  to 
perform  your  tasks  for  this 

0 

0 

0 

0 

0 

0 

0 

phase  of  the  battle? 

3.  Did  you  have  a  clear  picture  of 
the  battlefield  situation  for  the 

-3 

Extremely 

Unclear 

•2 

-1 

0 

Neutral 

+  1 

+2 

+3 

Extremely 

Clear 

Start  of  the  exercise? 

o 

0 

0 

0 

0 

0 

0 

4.  Did  the  DST  and  SYNCH 

-3 

-2 

-1 

0 

+l 

+2 

+3 

MATRIX  review  give  you  a 
clear  picture  of  what  the 

Extremely 

Unclear 

Neutral 

Extremely 

Clear 

commander  expected  during 
this  phase  of  the  battle? 

o 

0 

0 

0 

0 

0 

0 

5.  How  appropriate  was  the  length 
of  the  Staff  Preview? 

-3 

Far  Too 
Short 

-2 

-1 

0 

About 

Right 

+1 

+2 

+3 

Far  Too 
Long 

o 

0 

0 

0 

0 

0 

0 

6.  What  could  be  done  to  improve  the  Section  Preparation?  Comments? 
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Mini-Rehearsal 


1 .  How  effective  was  the 
Mini-Rehearsal  in  getting  the  staff 
ready  to  conduct  the  exercise? 

-3 

Extremely 

Ineffective 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Effective 

0 

0 

0 

0 

0 

0 

0 

2.  Did  you  have  a  clear  picture  of 
what  should  occur  according  to 
the  plan  during  the  exercise? 

-3 

Extremely 

Unclear 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Clear 

0 

0 

0 

0 

0 

0 

0 

3,  Did  you  have  a  clear  picture  of 
what  the  commander  and 

XO/BC  expected  from  your 
section  during  the  exercise? 

-3 

Extremely 

Unclear 

0 

-2 

0 

-1 

0 

0 

Neutral 

0 

+1 

0 

+2 

0 

+3 

Extremely 

Clear 

0 

4.  How  appropriate  was  the  length 
of  the  Mini-Rehearsal? 

-3 

Far  Too 
Short 

-2 

-1 

0 

About 

Right 

+1 

+2 

+3 

Far  Too 
Long 

0 

0 

0 

0 

0 

0 

0 

5.  What  could  be  done  to  improve  the  Mini-Rehearsal?  Comments? 
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Exercise  Execution 


1 .  Did  your  section  receive  the 

-3 

-2 

-1 

0 

+I 

+2 

+3 

message  traffic  you  would  have 

Far  Too 
Little 

About 

Right 

Far  Too 
Much 

expected  in  this  situation? 

0 

0 

0 

0 

0 

0 

0 

2.  Did  your  section  receive  the 

-3 

-2 

-1 

0 

+i 

+2 

+3 

information  you  would  have 

Far  Too 

About 

Far  Too 

expected  to  receive  in  this 

Little 

Right 

Much 

situation? 

0 

0 

0 

0 

0 

0 

0 

3-  In  this  exercise  you  received 

-3 

-2 

-1 

0 

+1 

+2 

+3 

only  messages  containing 

Definitely 

No 

Neutral 

Definitely 

accurate  information.  If  you 

I  Ca 

were  at  the  crawl  level  of 

proficiency  this  training  is 

0 

0 

0 

0 

0 

0 

0 

designed  for,  would  you  want  to 
receive  false  or  misleading 
information  that  would  require 
you  to  evaluate  each  message’s 
quality  and  accuracy? 


4.  Did  your  section  have  the  tools 
(overlays,  power  charts,  etc.)  it 
needed  for  the  exercise? 

-3 

Definitely 

No 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Definitely 

Yes 

0 

0 

0 

0 

o 

0 

0 

5.  What  could  be  done  to  improve  the  Exercise  Execution?  Comments? 
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Section  AAR  (SAAR) 


1 .  How  effective  was  the  SAAR 
material  in  guiding  your 
section’s  AAR? 

-3 

Extremely 

Ineffective 

-2 

-1 

0 

Neutral 

+  1 

+2 

+3 

Extremely 

Effective 

0 

0 

0 

0 

0 

0 

0 

2.  How  complete  was  the  SAAR 
materials  for  your  staff  section? 

-3 

Extremely 

Incomplete 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Complete 

0 

0 

0 

0 

0 

0 

0 

3.  How  effective  was  the  SAAR  in 
focusing  the  section’s  attention 
on  improving  the  learning 
objective: 

-3 

Extremely 

Ineffective 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Effective 

monitor  unit  operations? 

0 

0 

0 

0 

0 

0 

0 

process  information  and 
messages? 

0 

0 

0 

0 

0 

0 

0 

analyze  information? 

0 

0 

0 

0 

o 

0 

0 

communicate  mission  critical 
information? 

0 

0 

0 

0 

0 

0 

0 

coordinate  information  and 
intelligence? 

0 

0 

0 

0 

0 

0 

0 

integrate  staff  input? 

0 

0 

0 

0 

0 

0 

0 

recommend  a  course  of 
action? 

0 

0 

0 

0 

o 

0 

0 

disseminate  the  commander’s 
decision? 

0 

0 

0 

0 

0 

0 

0 
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4.  What  else  was  needed?  Comments? 
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Large  Group  AAR 


1 .  How  effective  was  the  Large 
Group  AAR  material  in  guiding 
your  AAR? 

-3 

Extremely 

Ineffective 

-2 

-1 

0 

Neutral 

+  1 

+2 

+3 

Extremely 

Effective 

0 

0 

0 

0 

0 

0 

0 

2.  How  complete  was  the  Large 
Group  AAR  materials  for  your 
staff? 

-3 

Extremely 

incomplete 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Complete 

0 

0 

0 

0 

0 

0 

0 

3.  How  effective  was  the  Large 
Group  AAR  in  focusing  your 
staffs  attention  on  improving 
their  performance  on  the 
learning  objective: 

-3 

Extremely 

Ineffective 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Effeaive 

analyze  information? 

0 

0 

0 

o 

o 

0 

0 

communicate  mission  critical 
information? 

0 

0 

0 

o 

0 

0 

0 

coordinate  information  and 
intelligence? 

o 

0 

0 

o 

0 

o 

0 

integrate  staff  input? 

0 

0 

0 

0 

0 

0 

0 

recommend  a  course  of 
action? 

0 

0 

0 

0 

0 

0 

0 

disseminate  the  commander’s 
decision? 

0 

0 

0 

0 

0 

0 

0 

4.  What  could  be  done  to  improve  the  Large  Group  AAR?  Comments? 
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Exercise  Evaluation 


1 .  Did  the  exercise  achieve  its 
objectives? 

-3 

Definitely 

No 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Definitely 

Yes 

0 

0 

0 

0 

0 

0 

0 

2.  In  your  opinion,  how  effective 
was  the  exercise? 

-3 

Extremely 

Ineffective 

-2 

-I 

0 

Neutral 

+1 

+2 

+3 

Extremely 

Effective 

0 

0 

0 

0 

0 

0 

0 

3.  Was  the  exercise  focused  on  the 
correct  tasks  for  your  staff 

-3 

Definitely 

No 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Definitely 

Yes 

section? 

0 

0 

0 

0 

0 

0 

0 

4.  Was  the  exercise  focused  on  the 
correct  tasks  for  a  maneuver 

-3 

Definitely 

No 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Definitely 

Yes 

brigade  staff? 

0 

0 

0 

0 

0 

0 

0 

5.  Did  the  exercise  present  a 
sufficiently  challenging 

-3 

Far  Too 

Easy 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Far  Too 
Hard 

experience  for  you? 

0 

0 

0 

0 

0 

0 

0 

6.  Did  the  exercise  present  a 
sufficiently  challenging 

-3 

Far  Too 

Easy 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Far  Too 
Hard 

experience  for  your  staff 

section? 

0 

0 

0 

0 

0 

0 

0 

7.  Did  the  exercise  present  a 
sufficiently  challenging 
experience  for  the  brigade 

-3 

Far  Too 

Easy 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Far  Too 
Hard 

staff? 

0 

0 

0 

0 

0 

0 

0 

8.  Comments: 
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9.  Did  the  exercise  improve  your 
section’s  ability  to  perform  the 
learning  objective: 

-3 

Definitely 

No 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Definitely 

Yes 

monitor  unit  operations? 

0 

0 

0 

0 

0 

0 

0 

process  information  and 
messages? 

0 

0 

0 

0 

0 

0 

0 

analyze  information? 

0 

0 

0 

0 

0 

0 

0 

communicate  mission  critical 
information? 

0 

0 

0 

0 

0 

0 

0 

10.  Did  the  exercise  improve  your 
staffs  ability  to  perform  the 
learning  objective: 

-3 

Definitely 

No 

-2 

-1 

0 

Neutral 

+1 

+2 

+3 

Definitely 

Yes 

coordinate  information  and 
intelligence? 

0 

o 

0 

0 

0 

0 

0 

integrate  staff  input? 

0 

o 

0 

0 

0 

0 

0 

recommend  a  course  of 
action? 

0 

0 

0 

0 

0 

0 

0 

disseminate  the  commander’s 
decision? 

0 

0 

0 

0 

0 

0 

0 

11.  What  else  was  needed?  Comments? 
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What  were  the  three  strongest  features  of  the  exercise? 


Q  There  were  no  weak  features. 
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APPENDIX  D 

KEY  TECHNOLOGY  INNOVATIONS  AND  DEVELOPMENT 


This  effort  focused  on  improving  and  refining  the  system  developed  for  SGT.  During  the 
previous  SGT,  “technology”  was  synonymous  with  the  software  code  originally  written  for  the 
CVCC  project  that  ran  on  the  Sun®  system.  The  refinements  called  for  in  the  current  effort 
required  a  broader  definition  of  “technology.”  This  project  refined  the  custom  C  code  for  the 
Sun  platforms  that  was  developed  under  the  previous  SGT  effort  but  applied  many  additional 
technology  innovations  to  meet  the  needs  of  the  users.  Intel®  based  PCs  were  added  to  assist  in 
staff  section  preparations,  WST,  the  display  of  training  aids  during  exercise  execution  and  the 
staff  section  AAR.  These  applications  used  MS  Visual  Basic  5®  and  MS®  Help  code  for  the 
computers  running  the  Windows95®  Operating  System.  The  Newton  2000®  PDAs  with  Quick 
Figure  spreadsheets  were  incorporated  to  assist  observers,  and  finally  the  exercise  preview  and 
the  exercise  AAR  were  developed  using  the  multimedia  program  Macromedia  Director®. 

Methodology 

The  initial  list  of  technology  modifications  was  developed  during  the  design  phase  of  the 
project  and  was  based  on  the  following:  (a)  shortcomings  in  the  system  which  emerged  during 
the  previous  SGT  formative  evaluation,  (b)  system  features  that  had  not  been  developed  earlier 
even  though  they  were  on  the  list  of  desired  software,  and  (c)  solutions  needed  if  the  training 
materials  were  to  be  modified  in  accordance  with  the  commanders’  needs.  The  nine  learning 
objectives  and  the  small  group  training  methodology  were  used  to  evaluate  whether  the 
technology  solutions  would  be  implemented.  If  the  technology  provided  no  training 
contributions,  then  it  was  not  implemented. 

As  each  technology  innovation  was  implemented,  the  project  development  team  ran  tests  to 
see  if  expected  performance  standards  were  being  met  and  if  the  innovation  appeared  to 
contribute  positively  to  the  training  environment.  If  no  problems  were  encountered  during  this 
stand-alone  trial,  the  development  team  integrated  and  tested  the  innovation  in  a  full  training 
environment.  During  the  external  formative  evaluation  events,  training  audience  problems  or 
concerns  were  recorded.  These  were  reviewed  by  the  Team  for  potential  modifications  or 
additions. 


Key  Technology  Innovations 

The  SGT  Team  organized  the  potential  technology  solution  implementation  process  into 
several  areas: 

1.  Technology  changes  that  would  simplify  system  operating  capabilities  or  improve 

instrumented  data  collection. 

2.  Modifications  that  would  simplify  the  job  of  the  exercise  administrator. 

3.  Tools  that  would  facilitate  training  for  trainees,  interactors,  and  observers. 

4.  Technology  changes  to  support  the  development  of  computerized  TSP  components. 
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System  Operations  and  Instrumentation  Data  Collection  Capabilities 


Simplified  File  Structure 

The  most  significant  change  made  in  this  area  involved  the  simplification  of  the  network  file 
structures  to  a  system  more  similar  to  a  client/server.  Under  the  previous  version  of  SGT  each  of 
the  computers  mounted  the  file  systems  of  all  the  other  systems  (a  peer/peer  system).  This  meant 
that  in  a  TOC  setup  with  four  player  workstations,  each  loaded  itself  and  the  other  three 
positions.  This  approach  made  it  extremely  difficult  to  operate  the  system  if  one  computer  failed 
to  boot.  It  was  also  an  inefficient  use  of  disk  space.  The  modification  changed  the  file  structures 
so  that  all  of  the  workstations  mounted  from  a  single  hard  drive  external  to  all  the  systems. 

Impacts  of  the  change.  There  were  several  improvements  resulting  from  the  simplified  file 
structure. 

1.  It  is  now  no  longer  necessary  to  boot  up  the  machines  in  a  rigid  order  or  even  boot  up  all 
the  machines. 

2.  It  is  easy  to  reconfigure  the  system  and  add  in  new  player  positions.  This  facilitates 
ease  of  operations  for  both  training  and  development. 

3.  The  developers  have  quick  access  to  files  used  on  the  workstations  and  can  troubleshoot 
modifications. 

Recommended  improvement.  The  network  protocols  should  also  be  updated.  This  would 
facilitate  the  migration  of  the  SGT  to  a  different  host  platform. 

Improved  Data  Collection 

The  previous  SGT  had  instrumentation  files  which  tracked  certain  actions  that  occurred  at  a 
workstation,  such  as  when  a  message  arrived  at  a  workstation,  when  it  was  opened,  when  a  new 
message  was  composed,  and  when  a  message  was  sent.  With  the  increased  integration  of  the 
learning  objectives  into  all  the  dimensions  of  the  training  project  it  was  also  necessary  to  collect 
data  on  the  actions  which  could  measure  behavior  associated  with  the  learning  objectives 
analysis  and  communicate.  The  capability  to  record  when  overlays  were  updated  at  the 
workstations  (analysis)  and  when  they  were  sent  to  the  TOC  map  display  (communication)  was 
therefore  added. 

Impact  of  the  change.  The  extended  instrumentation  permitted  the  Team  to  improve  the 
section  AAR  graphic  feedback  (staff  fingerprint  and  WOO)  to  the  staff  sections.  This  was  done 
by  linking  their  performance  to  more  complex  learning  objective  dimensions. 

Recommended  improvements.  The  present  instrumentation  will  only  collect  data  if  the 
action  is  performed  using  a  certain  procedure;  thus  if  a  trainee  develops  a  way  to  do  something 
more  efficiently,  the  machine  may  not  record  this  performance.  Modifications  could  ensure  that 
no  matter  how  the  operation  was  performed  only  the  result  would  be  recorded. 
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Exercise  Administration  Functions 


Exercise  Administrator  Workstation  Graphic  User  Interface 


In  the  prior  SGT  project,  it  was  necessary  to  spend  an  entire  day  training  the  exercise 
administrator.  Furthermore,  the  individual  had  to  have  a  good  understanding  of  computer  file 
structures  and  some  experience  with  UNIX  commands  prior  to  the  training.  If  this  individual 
accidentally  made  a  mistake  when  typing  an  entry,  it  was  possible  to  render  the  system  non- 
operational  until  the  software  could  be  reinstalled.  The  new  exercise  administrator  interface  has 
simplified  the  system  by  implementing  a  graphic  user  interface.  Only  one  UNIX  command  is 
needed  to  launch  the  system.  After  that  the  exercise  administrator  can  choose  to:  (a)  select  the 
exercise  vignette,  (b)  load  the  vignette,  (c)  start  all  workstations,  (d)  reset,  pause  and  resume  the 
game  clock,  or  (e)  recover  crashed  workstations. 

Impacts  of  the  change.  The  improved  workstation  graphic  user  interface  resulted  in  several 
benefits. 

1.  It  decreased  training  preparation  time  and  made  the  system  so  simple  to  use  that  many 
more  unit  personnel  can  meet  the  requirements  necessary  to  function  as  an  exercise 
administrator. 

2.  It  reduced  the  chances  that  the  exercise  administrator  will  cause  a  system  crash  during  an 
exercise. 

Recommended  improvements.  The  designers  should  incorporate  a  touch  screen  interface 
and  remove  the  keyboard  and  mouse  fi’om  the  terminal.  This  would  speed  exercise  administrator 
actions  and  minimize  the  possibility  that  the  trainer  could  accidentally  crash  the  system.  Without 
a  keyboard  or  mouse,  it  would  not  be  possible  to  make  UNIX  command  line  entries. 

System  Support  for  Trainees,  Interactors,  and  Observers 
Point  of  Training  Exercise  Support  Materials 

Previous  simulation  training  found  that  trainees  were  often  unable  to  find  the  correct  training 
materials  for  a  given  exercise  (due  to  the  excessive  size  of  the  TSP  documentation),  or  had  failed 
to  bring  the  correct  documents.  This  led  to  decreased  training  effectiveness.  This  effort 
remedied  the  problem  with  the  development  of  “point  of  training”  exercise  materials  printed  on 
location.  With  a  click  of  the  mouse  on  a  labeled  desktop  icon,  a  custom  Visual  Basic  5®  program 
running  on  a  Windows  95®  based  personal  computer  executed  an  exercise  unique  print  program. 
All  mandatory  materials  were  produced,  neatly  labeled  and  collated  for  use. 

Impacts  of  the  change.  This  change  provides  several  benefits. 

1.  All  individuals  participating  in  the  training  —  trainees,  observers,  interactors,  and 
trainers  —  were  provided  with  the  printed  information  essential  for  the  successful 
accomplishment  of  exercises. 
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2.  This  would  reduce  the  amount  of  materials  stored  by  the  unit  and  minimize  the 
possibility  that  essential  training  materials  would  be  lost. 

Recommended  improvements.  None. 

Simplified  Map  Commands 

The  map  for  the  previous  SGT  effort  was  so  difficult  to  use  that  operations  which  should 
have  been  conducted  by  the  sections  were  not  done.  The  workstation  operators  could  not  figure 
how  to  accomplish  this  during  the  one  hour  exercise.  Some  of  the  most  serious  shortcomings 
with  the  earlier  system  and  the  SGT  remedies  for  those  shortcomings  are  shovm  in  Table  D1 . 

Table  D1 

Map  Command  Shortcomings  and  Associated  Improvements 


Initial  Shortcomings 

Improvements 

Poorly  structured  menus  caused  confusion. 

Restructured  menus  more  closely  match  military  concepts  of 
map/overlay  operations. 

Drawing  map  icons  was  difficult  and  time 
consuming. 

Library  of  commonly  used  unit  icons  developed.  These  unit 
icons  were  placed  on  the  desktop  and  could  be  used  by 
simply  dragging  and  dropping.  The  set  of  library  icons 
displayed  at  each  workstation  represented  those  icons  most 
commonly  used  by  that  section  (i.e.,  EN  -  mines  symbols, 

FSE  “  artillery  units).  Any  symbols  or  icons  developed  by 
the  training  participants  were  added  to  the  desktop  for  use 
again  at  any  time  during  the  exercise. 

A  section  could  edit  any  overlay  and  send  it  to 
the  TOC  SIT  DISPLAY  (e.g.,  EN  could  edit 

Overlay  ownership  was  established  at  each  workstation.  All 
the  workstations  could  view  any  overlay;  however,  each 

S-2  overlay  and  send  to  TOC  SIT  DISPLAY). 

section  could  modify  only  their  own  overlays. 

Message  icons  were  sent  straight  to  the  SIT 
DISPLAY  without  analysis,  causing  that 
display  to  become  cluttered  since  removing 
them  was  done  infrequently  or  never. 

Message  icons,  if  sent  to  the  SIT  DISPLAY,  blink  to  draw 
attention  to  the  update  but  cease  to  display  after  90  seconds. 

If  they  are  important  they  must  be  added  to  the  section 
overlay  and  that  overlay  is  used  to  update  the  SIT 

DISPLAY. 

Workstation  operator  had  difficulty  knowing 
when  he  was  entering  and/or  leaving  the  MAP 
Edit  mode. 

Overlays  belonging  to  a  workstation  (i.e.,  those  which  can 
be  modified)  are  always  in  edit  mode.  All  other  overlays 
can  only  be  viewed. 

100  km  X  100  km  map  size  limited  ability  of 
the  brigade  staff  to  track  division  operations. 

Size  increased  to  200  km  X  200  km. 

Impacts  of  the  changes.  The  simplified  map  commands  resulted  in  improved  operations. 

1 .  Workstation  operators  required  less  time  to  become  proficient  with  map  editing. 

2.  Fewer  errors  were  made  with  the  map  overlays. 

3.  The  technology  became  less  of  a  training  distracter. 

Recommended  improvements.  There  were  two  recommended  improvements  that  could  be 
made. 
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1 .  Add  an  8x  power  zoom  (in  and  out)  for  viewing  the  map.  This  would  help  the  training 
audience  scale  the  map  more  efficiently. 

2.  Develop  “help  wizards”  to  do  map  editing  tasks. 

Automated  Observer  Checklists  and  Coaching  Questions 

In  the  previous  SGT  project,  there  was  one  observer  for  each  workstation.  These  were 
trained,  standing  0/C  team  observers.  They  received  printed  checklists  and  coaching  questions 
linked  to  the  critical  messages  to  be  sent  by  the  system  during  the  exercise.  During  the  pilots  and 
the  trials  some  of  the  observers  used  the  checklists  and  coaching  questions  while  others  did  not. 
One  complaint  concerned  tracking  the  game  time.  If  the  observer  lost  track  of  the  game  time, 
coaching  opportunities  were  either  late  or  missed  completely.  This  project  was  designed  to  have 
two  untrained  observers  in  the  main  CP  to  observe  the  BC/XO  and  the  four  staff  sections. 
Successful  execution  of  training  required  that  these  observers  have  tools  which  would  structure 
their  observations  in  much  the  same  manner  as  the  structured  message  traffic  ensured  that  the 
training  audience  was  exposed  to  the  timely  and  correct  tactical  trigger  events.  The  team 
programmed  PDAs  with  checklists  that  were  time  sequenced  and  asked  questions  about  staff 
actions  which  could  be  observed  directly  and  answered  with  a  “1”  for  observed  and  left  blank  if 
not  observed.  These  observations  were  related  to  either  the  timeliness  or  the  accuracy  of  a 
specific  learning  objective.  Coaching  questions  and  directions  on  who  to  monitor  and  what 
should  be  going  on  appeared  on  the  screen,  synchronized  with  the  game  clock  and  the  message 
stream. 

Impacts  of  the  change.  This  change  had  several  beneficial  results. 

1.  The  automation  of  the  checklist  along  with  alarms  and  coaching  questions  permitted 
successful,  effective  observation  by  less  experienced  personnel. 

2.  The  data  collected  on  the  PDAs  was  immediately  available  to  the  commander  on-site.  He 
could  use  this  data  to  evaluate  what  had  just  occurred  in  a  given  section  during  the 
exercise  while  the  staff  sections  conducted  their  section  AAR  and  self-assessment. 

Recommended  improvements.  None. 

Interactor  Workstation  Improvements 

During  the  previous  SGT  effort,  the  Team  observed  that  the  interactor  positions  were 
manned  by  personnel  who  used  the  mouse  and  keyboard  with  some  difficulty.  These  individuals 
were  handicapped  in  providing  the  necessary  responses  to  an  RFI  coming  fi-om  the  training 
audience.  Several  improvements  were  made  in  the  white  cell  workstations  which  improved  the 
speed  and  accuracy  with  which  they  could  react.  Touch  screens  were  added  to  the  message 
display  screen.  Entries  that  were  selected  fi-om  the  pull  down  menus  could  be  done  with  a  mouse 
click.  In  addition,  private  overlays  were  established  for  the  Higher  and  Adjacent  Command  and 
Control  (HACC)  and  Subordinate  and  Adjacent  Command  and  Control  (SACC).  The  trainees 
could  not  post  these  overlays,  but  they  permitted  the  HACC  and  SACC  to  see  the  current  ground 
truth  overlay  every  five  minutes  during  the  exercise. 
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Impact  of  the  change.  The  enhanced  workstations  improved  white  cell  responsiveness. 

Recommended  improvements.  The  design  should  require  that  the  interactors  participate  in  a 
WST  session  tailored  for  white  cell  operations. 

Train-the-Trainer  Materials 


In  the  prior  SGT,  all  of  the  T^  materials  were  printed.  In  an  effort  to  meet  the  demands  of 
the  commanders  to  make  the  SGT  exportable  and  easily  trained,  the  Team  developed  a 
workstation  operator  tutorial  using  MS  Help®.  This  application  operated  exaetly  like  the  help 
packages  associated  with  Microsoft  Office  95®;  however,  it  was  customized  to  contain 
information  on  how  to  perform  the  operations  necessary  to  serve  as  a  workstation  operator  during 
an  exercise. 

Impact  of  the  change.  On-site  workstation  training  was  cut  from  four  hours  to  30  minutes. 

Recommended  improvements.  One  improvement  would  be  to  place  all  other  T^  packages  in 
MS  Help®  custom  applications  and  eliminate  printed  materials. 

Structured  Section  AARs 

This  SGT  effort  resulted  in  two  different  types  of  AARs.  The  first  one — ^the  section  AAR — 
was  designed  to  run  in  Windows  95*  and  was  written  in  Visual  Basic  5®.  The  section  AAR  used 
information  collected  by  the  computer  instrumentation  package  and  presented  it  graphically  to 
the  section,  along  with  a  set  of  self-assessment  questions.  These  questions  focused  on  staff 
section  performance  in  response  to  specific  critical  event  triggers.  When  completed,  the  section 
was  provided  a  printed  copy  of  the  ratings  they  gave  themselves,  the  rationale  for  the  rating,  and, 
if  the  ratings  were  substandard,  a  list  of  remediation  sources. 

Impact  of  the  change.  The  section  self-assessments  had  positive  impacts  on  the  seetion 
AAR. 

1.  This  information  permitted  the  section  to  evaluate  their  own  performance  on  each  of  the 
key  critical  specific  performance  measures. 

2.  The  section  AAR  could  be  conducted  with  no  assistance  from  an  observer. 

Recommended  improvements.  The  design  should  improve  the  content  of  the  remediation 
data  printed  by  expanding  to  a  complete  paragraph  instead  of  a  single  line  citing  a  reference. 


TSP  Development  Tools 


Exercise  Development  Tools 

Improvements  have  been  made  to  one  of  the  programs  developed  during  the  previous  effort 
to  make  message  input  simpler,  more  rapid,  and  more  accurate.  (Message  input  changes  can  now 
be  done  with  a  simple  edit  menu  instead  of  using  the  UNIX  VI  text  editor.)  With  this  new  tool. 
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most  of  the  files  needed  to  change  exercises  can  be  created  or  updated  with  ease.  The  tool  uses  a 
graphic  user  interface  (and  pull  down  menus  in  places  where  limited  input  is  required)  to  ensure 
that  new  or  modified  files  will  be  in  the  correct  format. 

Impact  of  the  change.  Exercises  can  be  quickly  developed  or  modified  as  needed  with 
minimal  effort  or  manpower. 

Recommended  improvements.  Develop  a  tool  system  that  simplifies  the  development  of 
exercise  overlays. 
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APPENDIX  E 

PILOT  AND  TRIAL  SCHEDULES 


Table  El 
Pilot  schedule 


Day/Time 

Event 

Audience 

Saturday 

0900-0930 

Orientation 

All 

0930-1000 

OPORDs  ReviewAVorkstation  Operators 

Training 

All 

1000-1300 

Execute  Staff  Transition  Table 

All 

1300-1330 

Lunch 

All 

1330-1430 

Table  Hot  Wash 

All 

1430-1830 

Execute  Staff  Integration  Table 

All 

1830-1930 

Table  Hot  Wash 

All 

Sunday 

0700-1200 

Execute  Command  Post  Table 

All* 

1200-1330 

Table  and  Program  Hot  Wash 

All 

♦Note:  The  EN  staff  section  is  not  required  to  participate  in  this  exercise 

Table  E2 

Trial  schedule 

Day/Time 

Event 

Audience 

Friday 

1800-2000 

OPORDs  ReviewAVorkstation  Operators 

Training 

All 

Saturday 

0800-0830 

Workstation  Familiarization  Exercise/Officer-in- 
Charge  Orientation 

All 

0830-1100 

Execute  Staff  Transition  Table 

All 

1100-1200 

Lunch 

1200-1600 

Execute  Staff  Integration  Table 

All 

1600-1700 

End  of  Day  Hot  Wash 

All 

Sunday 

0800-1200 

Execute  Command  Post  Table 

All* 

1200-1300 

Final  Hot  Wash 

All 

1300-1330 

Site  Recovery/Departure 

All 

♦Note:  The  EN  staff  section  is  not  required  to  participate  in  this  exercise. 

E-1 


